{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "1_dqn_kearas_cartpole.ipynb",
      "version": "0.3.2",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python (spinningup)",
      "language": "python",
      "name": "spinningup"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XT2oM6FHVJCf"
      },
      "source": [
        "## Enviroment:\n",
        "\n",
        "Open AI gym [CartPole v0](https://github.com/openai/gym/wiki/CartPole-v0)\n",
        "\n",
        "### Observation\n",
        "\n",
        "Type: Box(4)\n",
        "\n",
        "| Num  | Observation          | Min      | Max     |\n",
        "| :--- | :------------------- | :------- | :------ |\n",
        "| 0    | Cart Position        | -2.4     | 2.4     |\n",
        "| 1    | Cart Velocity        | -Inf     | Inf     |\n",
        "| 2    | Pole Angle           | ~ -41.8° | ~ 41.8° |\n",
        "| 3    | Pole Velocity At Tip | -Inf     | Inf     |\n",
        "\n",
        "### Actions\n",
        "\n",
        "Type: Discrete(2)\n",
        "\n",
        "| Num  | Action                 |\n",
        "| :--- | :--------------------- |\n",
        "| 0    | Push cart to the left  |\n",
        "| 1    | Push cart to the right |\n",
        "\n",
        "Note: The amount the velocity is reduced or increased is not fixed as it depends on the angle the pole is pointing. This is because the center of gravity of the pole increases the amount of energy needed to move the cart underneath it\n",
        "\n",
        "### Reward\n",
        "\n",
        "Reward is 1 for every step taken, including the termination step\n",
        "\n",
        "### Starting State\n",
        "\n",
        "All observations are assigned a uniform random value between ±0.05\n",
        "\n",
        "### Episode Termination\n",
        "\n",
        "1. Pole Angle is more than ±12°\n",
        "2. Cart Position is more than ±2.4 (center of the cart reaches the edge of the display)\n",
        "3. Episode length is greater than 200\n",
        "\n",
        "### Solved Requirements\n",
        "\n",
        "Considered solved when the average reward is greater than or equal to 195.0 over 100 consecutive trials"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xzVWtGxJVJCh"
      },
      "source": [
        "## 1. gym enviroment setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "G312rzBjVJCi",
        "outputId": "edbd06e3-9424-4033-893f-0cef8e232a93",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import gym\n",
        "import numpy as np\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "env = gym.make(\"CartPole-v0\")\n",
        "env.reset()"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-0.03753636, -0.01905513,  0.03099227, -0.04468279])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "80YCNcHUZYtI",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import models\n",
        "from tensorflow.keras import losses,optimizers,metrics\n",
        "from tensorflow.keras.layers import Dense, Activation, Flatten\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from collections import deque\n",
        "import random\n",
        "from tqdm import tqdm"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IjkusASudj9W",
        "colab": {}
      },
      "source": [
        "ACTION_SPACE_SIZE = env.action_space.n\n",
        "REPLAY_MEMORY_SIZE = 50_000\n",
        "MIN_REPLAY_MEMORY_SIZE = 1_000\n",
        "MINIBATCH_SIZE = 64\n",
        "UPDATE_TARGET_EVERY = 5\n",
        "DISCOUNT = 0.99\n",
        "EPISODES =1000\n",
        "\n",
        "# Exploration settings\n",
        "epsilon = 1  # not a constant, going to be decayed\n",
        "EPSILON_DECAY = 0.995\n",
        "MIN_EPSILON = 0.001\n",
        "\n",
        "ep_rewards = []\n",
        "\n",
        "AGGREGATE_STATS_EVERY = 50\n",
        "MIN_EPSILON = 0.001"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k4-a2aU0YCbL"
      },
      "source": [
        "### Plot epsilon decay "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "41yF0b4u46gb",
        "outputId": "1cd1b940-84cd-492f-8597-b8414dfc195c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "recorder = {\"epsode\":[],\"epsilon\":[]}\n",
        "\n",
        "for epsode in range(EPISODES):\n",
        "    if epsilon > MIN_EPSILON:\n",
        "        epsilon *= EPSILON_DECAY\n",
        "        epsilon = max(MIN_EPSILON, epsilon)\n",
        "    \n",
        "    recorder[\"epsode\"].append(epsode)\n",
        "    recorder[\"epsilon\"].append(epsilon)\n",
        "\n",
        "plt.plot(recorder[\"epsode\"], recorder[\"epsilon\"])"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f4b19c8f438>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8VOWdx/HPb2YyuZMQEgIkgXAJ\nSkRAGi+ILF2v6Cq23l27arV1bWutbV/tanfrdt19rdt1V2u71mqtbW2t1norRVu81NZLUQwKyEUg\n3AOYhFsSQq4zz/4xAwaKJoRJTubM9/16zYs5z3mS+Z0c/ebkOec8x5xziIiIvwS8LkBERBJP4S4i\n4kMKdxERH1K4i4j4kMJdRMSHFO4iIj6kcBcR8SGFu4iIDyncRUR8KOTVBxcWFrry8nKvPl5EJCkt\nXrx4h3OuqKd+noV7eXk51dXVXn28iEhSMrNNvemnYRkRER9SuIuI+JDCXUTEhxTuIiI+1GO4m9nD\nZlZvZss/Yr2Z2ffNrMbMlpnZ9MSXKSIiR6I3R+4/A+Z8zPpzgYr46wbg/qMvS0REjkaP4e6cexXY\n9TFdLgQecTFvAvlmNjJRBYqIyJFLxJh7CbCl23JtvO2vmNkNZlZtZtUNDQ19+rB3Nu/mu394Hz0e\nUETkow3oCVXn3IPOuSrnXFVRUY83WB3W8q2N3P+ndaxr2Jvg6kRE/CMR4b4VKOu2XBpv6xdnVRYD\nsGBFXX99hIhI0ktEuM8Dro5fNXMK0Oic256A73tYI/MymVqaxwsrPuivjxARSXq9uRTyMWAhcIyZ\n1ZrZ9WZ2o5ndGO/yPLAeqAF+DHyx36qNO/u4ESytbWR7Y2t/f5SISFLqceIw59yVPax3wJcSVlEv\nnHPcCO5asJoXV9Zx9YzygfxoEZGkkJR3qE4YnsO4omwWaGhGROSwkjLcIXb0/ub6XezZ1+F1KSIi\ng07ShvvZlcVEoo4/vl/vdSkiIoNO0ob71NJ8ioeka2hGROQwkjbcAwHjrMpi/rymgdaOiNfliIgM\nKkkb7hAbd2/rjPLa2r5NZSAi4ldJHe4njx1GbkZId6uKiBwiqcM9HApw1qRiXlz5AR1dUa/LEREZ\nNJI63AH+bspImtq6eKNmh9eliIgMGkkf7qdVFJKbEWL+sn6bzkZEJOkkfbinh4KcXTmCFzQ0IyJy\nQNKHO8DfTRlBc1sXr9foqhkREfBJuJ82oUhDMyIi3fgi3MOhAOccN4IXV9TR3qUbmkREfBHuELtq\nprm9i9fX6qoZERHfhPvM8YXkZabxnIZmRET8E+7hUICzK4t5caWGZkREfBPu8OHQzGtrNDQjIqnN\nV+E+c0Ih+Vlp/G7ZNq9LERHxlK/CPS0Y4NzJI3lhRR0t7V1elyMi4hlfhTvAp6aNorUzwosrNVOk\niKQu34X7ieUFlORn8uySrV6XIiLiGd+FeyBgXDhtFK+t3cGOve1elyMi4gnfhTvAp04oIRJ1zF+q\nE6sikpp8Ge4Ti3OpHDmEZ5Yo3EUkNfky3AE+dcIolm7Zw4YdLV6XIiIy4Hwb7nOnlmAGz76rE6si\nknp8G+4j8jI4dfwwnl2yFeec1+WIiAwo34Y7wIXTSti0cx9LtuzxuhQRkQHl63CfM3kE6aEAz2ho\nRkRSjK/DfUhGGmcfN4LfLtlGW6dmihSR1NGrcDezOWa22sxqzOzWw6wfbWavmNm7ZrbMzM5LfKl9\nc1lVKY2tnby0StMRiEjq6DHczSwI3AecC1QCV5pZ5SHd/gV4wjl3AnAF8MNEF9pXp44vpCQ/kyeq\na70uRURkwPTmyP0koMY5t9451wE8Dlx4SB8HDIm/zwMGzd1DwYBx8fQSXlvbwLY9rV6XIyIyIHoT\n7iXAlm7LtfG27r4DfMbMaoHngS8f7huZ2Q1mVm1m1Q0NDX0ot28u+UQZzsHT7+joXURSQ6JOqF4J\n/Mw5VwqcB/zCzP7qezvnHnTOVTnnqoqKihL00T0bPSyLGeOG8UR1LdGornkXEf/rTbhvBcq6LZfG\n27q7HngCwDm3EMgAChNRYKJcWlXK5l37WLRxl9eliIj0u96E+9tAhZmNNbMwsROm8w7psxk4A8DM\nJhEL94Ebd+mFcyePJCc9xG90YlVEUkCP4e6c6wJuAhYAq4hdFbPCzO4ws7nxbl8HPm9mS4HHgGvd\nILvnPzMc5IKpo3j+ve00t3V6XY6ISL8K9aaTc+55YidKu7fd3u39SmBmYktLvEurSnls0WaeW7ad\nK04a7XU5IiL9xtd3qB7qhLJ8Kobn8NiizV6XIiLSr1Iq3M2Mq04ezdLaRt6rbfS6HBGRfpNS4Q7w\n6emlZKYF+dWiTV6XIiLSb1Iu3PMy05g7dRTPvruNJp1YFRGfSrlwB7jqlNG0dkb0lCYR8a2UDPcp\npfkcX5LHo29u1lOaRMSXUjLcAT5zymhW1zVTvWm316WIiCRcyob7BVNHkZse4tE3dWJVRPwnZcM9\nKxziouklPP/eB+xq6fC6HBGRhErZcAe46pQxdESiPFG9pefOIiJJJKXDfWJxLiePLeAXCzfRFYl6\nXY6ISMKkdLgDXHfaWLbuaeXFlXrGqoj4R8qH+5mTiikryOSnb2z0uhQRkYRJ+XAPBoxrZpSzaOMu\nlm/VfDMi4g8pH+4Al51YRnY4yMNvbPC6FBGRhFC4A0My0rjkE6XMX7qdhuZ2r8sRETlqCve4a04t\npyMS5dG3dFOTiCQ/hXvcuKIcTj92OL98cxPtXRGvyxEROSoK924+O7OcHXs7+N3S7V6XIiJyVBTu\n3Zw2oZBjinP58avrNVukiCQ1hXs3ZsY/zh7H6rpmXlld73U5IiJ9pnA/xAVTRzEqL4Mf/Xm916WI\niPSZwv0QacEA188ax6INu3hns+Z6F5HkpHA/jCtOLCMvM40f/Wmd16WIiPSJwv0wstNDXDNjDC+u\nqqOmfq/X5YiIHDGF+0e4+tRywsEAD76qo3cRST4K949QmJPOZVVlPPPuVj5obPO6HBGRI6Jw/xif\nnzWOSNTx49d05YyIJBeF+8cYPSyLT00r4dG3NmlCMRFJKgr3Htx0+gQ6uqI8pKN3EUkiCvcejCvK\nYe7UUTyycBM79+roXUSSQ6/C3czmmNlqM6sxs1s/os9lZrbSzFaY2a8SW6a3bjq9grauCD9+TQ/z\nEJHk0GO4m1kQuA84F6gErjSzykP6VAC3ATOdc8cBt/RDrZ6ZMDyHC6aM4pGFG9nV0uF1OSIiPerN\nkftJQI1zbr1zrgN4HLjwkD6fB+5zzu0GcM75btatL58+gdbOCD95XWPvIjL49SbcS4At3ZZr423d\nTQQmmtkbZvammc053DcysxvMrNrMqhsaGvpWsUcqinM57/iR/Pwvm9izT0fvIjK4JeqEagioAD4J\nXAn82MzyD+3knHvQOVflnKsqKipK0EcPnJtPr6Clo4sHX9XRu4gMbr0J961AWbfl0nhbd7XAPOdc\np3NuA7CGWNj7yjEjcpk7dRQ/fWMj9U26a1VEBq/ehPvbQIWZjTWzMHAFMO+QPs8SO2rHzAqJDdP4\n8vD2a2dNpDMS5Qd/rPG6FBGRj9RjuDvnuoCbgAXAKuAJ59wKM7vDzObGuy0AdprZSuAV4BvOuZ39\nVbSXxgzL5vITy3hs0WY279zndTkiIodlXj0rtKqqylVXV3vy2UerrqmN2Xe9wrmTR3LP5dO8LkdE\nUoiZLXbOVfXUT3eo9kHxkAyuPXUszy7ZyvsfNHldjojIX1G499EXZo8nJz3E/yxY7XUpIiJ/ReHe\nR3lZadw4ezwvraqneuMur8sRETmIwv0ofHZmOcNz0/mP51YRjXpz7kJE5HAU7kchKxziG+ccw5It\ne/jdsm1elyMicoDC/ShdPL2U40YN4b//sJq2zojX5YiIAAr3oxYIGP/yd5Vs3dPKT17XlMAiMjgo\n3BNgxvhhnF1ZzA9fqaG+WdMSiIj3FO4J8q3zJtERiXL3C2u8LkVEROGeKOWF2Vwzo5xfV29h5Tbd\n2CQi3lK4J9CXT69gaFaY23+7HK+mdRARAYV7QuVlpXHrucdSvWk3T79z6KzIIiIDR+GeYJdML2X6\n6Hzu/P0qGls7vS5HRFKUwj3BAgHjjgsns6ulg3te1MlVEfGGwr0fTC7J4x9OGcMjCzeyYluj1+WI\nSApSuPeTr519DEOzwnz72eWad0ZEBpzCvZ/kZaZx23mTeGfzHn5dvcXrckQkxSjc+9HF00s4ZVwB\n//n8Kur0QG0RGUAK935kZtx50RQ6uqLc/tvlXpcjIilE4d7PxhZmc8uZE1mwoo4/LN/udTkikiIU\n7gPgc7PGUjlyCN/+7Qoa9+nadxHpfwr3AZAWDPDfl0xhV0sHd/5+ldfliEgKULgPkMkleXxu1lge\nf3sLr6/d4XU5IuJzCvcB9NUzJzKuKJtvPLlUUxOISL9SuA+gjLQgd182jfrmdv7tdyu8LkdEfEzh\nPsCmleXzpb+dwNPvbNXVMyLSbxTuHvjy6ROYXDKEbz2znIbmdq/LEREfUrh7IC0Y4J7LprG3vYtv\nPfOeHuwhIgmncPdIRXEu3zznGF5cWcdjizT3jIgklsLdQ9fNHMusikL+7XcrWFPX7HU5IuIjCncP\nBQLG3ZdNIzcjjZt+9Q6tHRGvSxIRn+hVuJvZHDNbbWY1Znbrx/S72MycmVUlrkR/K8pN5+7LprKm\nbi93zF/pdTki4hM9hruZBYH7gHOBSuBKM6s8TL9c4CvAW4ku0u/+ZmIRN84ez2OLNjN/2TavyxER\nH+jNkftJQI1zbr1zrgN4HLjwMP3+HfguoInL++DrZ09kWlk+tz31Hpt37vO6HBFJcr0J9xKg++Uc\ntfG2A8xsOlDmnHvu476Rmd1gZtVmVt3Q0HDExfpZWjDAD648AQy++KvFtHVq/F1E+u6oT6iaWQC4\nG/h6T32dcw8656qcc1VFRUVH+9G+U1aQxfcun8byrU388zPLdf27iPRZb8J9K1DWbbk03rZfLjAZ\n+JOZbQROAebppGrfnDGpmJvPqOCpd2r55VubvS5HRJJUb8L9baDCzMaaWRi4Api3f6VzrtE5V+ic\nK3fOlQNvAnOdc9X9UnEKuOWMCv72mCLu+N0KFm/a7XU5IpKEegx351wXcBOwAFgFPOGcW2Fmd5jZ\n3P4uMBUFAsb3Lj+BkXmZfPHRxdQ36xy1iBwZ82pct6qqylVX6+D+46zc1sRF97/B8SV5/PJzJ5Me\nCnpdkoh4zMwWO+d6HPbWHaqDWOWoIdx1yVTe3rib257WBGMi0nshrwuQj3fB1FGsb2jhnpfWML4o\nhy/97QSvSxKRJKBwTwI3nzGB9Tv2cteC1YwtzOa840d6XZKIDHIalkkCZsZ3L57C9NH5fO2JJSzd\nssfrkkRkkFO4J4mMtCAPXl1FYU46n3ukmi27NEWBiHw0hXsSKcxJ56fXnkh7Z4SrH17Ezr16RJ+I\nHJ7CPclUFOfy8LUnsm1PK9f97G1a2ru8LklEBiGFexKqKi/gvr+fzvJtTdz4y8V0dEW9LklEBhmF\ne5I6s7KYOz99PK+t3cE3nlxKNKpr4EXkQ7oUMolddmIZDXvbuWvBanIzQvz7hZMxM6/LEpFBQOGe\n5L74yfE0tXXywJ/XEw4G+fb5kxTwIqJwT3Zmxq1zjqWjK8rDb2wgHArwT3OOUcCLpDiFuw+YGbef\nX0lnJMqP/ryO9FCAr5410euyRMRDCnefMDPumDuZjq4o9768lmDA+PLpE3QEL5KiFO4+EggYd140\nhUgU7n5xDa2dEb55joZoRFKRwt1nggHjrkumkJEW4P4/raO1I8Lt51cSCCjgRVKJwt2HAgHjPz41\nmaxwkB+/toHWjgj/edHxBBXwIilD4e5TZsa3zptEZjjE919ey77OCP976VTCId23JpIKFO4+ZmZ8\n7ayJZIWD/Nfv32fPvg5+eNV0cjPSvC5NRPqZDuNSwI2zx3PXJVP4y7qdXP7Am9Q16YHbIn6ncE8R\nl1aV8ZNrqti4s4WLfvgXauqbvS5JRPqRwj2FfPKY4fz6hhm0d0W5+P6FLNqwy+uSRKSfKNxTzPGl\neTzzxVMZlh3mqofe5Im3t3hdkoj0A4V7CioryOKZL87k5LHD+OZTy/j3+SvpimhOeBE/UbinqLys\nNH722RO59tRyfvL6Bq77eTWNrZ1elyUiCaJwT2GhYIDvzD2OOy86noXrdvDp+95gbZ1OtIr4gcJd\nuPKk0Tz6uVNoautk7v+9wbPvbvW6JBE5Sgp3AeCksQU8d/Msji/J45ZfL+G2p9+jrTPidVki0kcK\ndzmgeEgGv/r8ydw4ezyPLdrMxff/hU07W7wuS0T6QOEuBwkFA9x67rE8dHUVW3bt4/wfvM5vl2iY\nRiTZKNzlsM6sLOa5m2dRMTyHrzy+hK88/q6uphFJIr0KdzObY2arzazGzG49zPqvmdlKM1tmZi+b\n2ZjElyoDrawgiyf+cQZfO2si85dt59zvvcrCdTu9LktEeqHHcDezIHAfcC5QCVxpZpWHdHsXqHLO\nTQGeBP470YWKN0LBADefUcFTXziV9LQgf//Qm9z5/CqdbBUZ5Hpz5H4SUOOcW++c6wAeBy7s3sE5\n94pzbl988U2gNLFlitemleXz3M2nceVJo3ng1fWcd+9rmptGZBDrTbiXAN0nIKmNt32U64HfH26F\nmd1gZtVmVt3Q0ND7KmVQyAqH+M9PH88vrz+ZjkiUyx5YyLefXU5zm8biRQabhJ5QNbPPAFXAXYdb\n75x70DlX5ZyrKioqSuRHywA6raKQF776N1w3cyy/fGsT59zzKn98v87rskSkm96E+1agrNtyabzt\nIGZ2JvDPwFznXHtiypPBKisc4vYLKnnqC6eSnR7iup9V8/lHqtmya1/PXywi/a434f42UGFmY80s\nDFwBzOvewcxOAB4gFuz1iS9TBqvpo4fy3M2z+Kc5x/JGzQ7OvPvP3PvSWp1wFfFYj+HunOsCbgIW\nAKuAJ5xzK8zsDjObG+92F5AD/MbMlpjZvI/4duJD4VCAL3xyPC9/fTZnVhZzz0trOPueV3lpZR3O\nOa/LE0lJ5tX/fFVVVa66utqTz5b+9UbNDv513gpq6vdy6vhhfOu8SUwuyfO6LBFfMLPFzrmqnvrp\nDlVJuJkTCnn+5ll854JKVm1v4vwfvM4tj7+r8XiRAaQjd+lXTW2d/OhP6/jJ6xtwDq6dWc6Ns8dT\nkB32ujSRpNTbI3eFuwyI7Y2t/O8La3jqnVqy0oJcc2o5n581jqEKeZEjonCXQWltXTP3vryW597b\nTlZakGtnxkI+P0shL9IbCncZ1NbEQ/7597aTHQ5x9YwxXDuznOG5GV6XJjKoKdwlKaz+oJnv/zEW\n8mmBABdNL+Fzs8YxYXiO16WJDEoKd0kqG3e08NDr6/lNdS3tXVHOnFTMjbPHUVVe4HVpIoOKwl2S\n0o697TyycBO/WLiR3fs6mVKaxz+cMoYLpo4iIy3odXkinlO4S1Lb19HFU4tr+fnCTdTU7yU/K43L\nqsq46uTRjBmW7XV5Ip5RuIsvOOd4c/0ufvHmRhasqCPqHLMnFnF5VRlnTComHNJ9eJJaFO7iOx80\ntvHYos08/vZm6praGZqVxoXTSri0qpTjRml6A0kNCnfxra5IlNdqdvDk4lpeXFFHRyTKpJFDuPQT\npVwwdRRFuelelyjSbxTukhL27Otg3tJtPLm4lmW1jQQMThk3jAumjmLOcSN0B6z4jsJdUs6aumbm\nL93G/GXbWb+jhVDAmDmhkPOnjOTs40aQl5nmdYkiR03hLinLOceKbU3MX7ad+cu2Ubu7lVDAOHlc\nAWdOKubMScWUFWR5XaZInyjcRYgF/dLaRhas+ICXVtaxtn4vAMeOyOWsyljQH1+SRyBgHlcq0jsK\nd5HD2LCjhZdX1fHCyjqqN+4i6mBYdpjTKgqZVVHErIpCiodofhsZvBTuIj3Y3dLBK6vreXVNA6/X\n7GDH3g4AjinOZVZFIbMmFnFi+VCywiGPKxX5kMJd5AhEo45VHzTx2todvL52B4s27qKjK0ooYBxf\nmsdJYws4eWwBnxhToBOz4imFu8hRaO2IsGjjLt5av5NFG3axtHYPnRGHGUwaMYSTxhZw0tgCThid\nz4ghGZhpzF4GhsJdJIHaOiO8u3kPizbsYtHGnSzetJu2zigAw3PTmVaWz7TR+Uwrzef40jxyM3R0\nL/2jt+GuwUSRXshICzJj/DBmjB8GVNDRFWXFtkaWbtnDki17WFrbyAsr6wAwg4rhOUwpzee4UUOY\nNDL20nCODCSFu0gfhEMBThg9lBNGDz3QtmdfB0trG1myeQ9La/fwp9X1PLm49sD60qGZVMaDvnLU\nECpHDqF0aKaGdKRfKNxFEiQ/K8zsiUXMnlh0oK2+uY2V25pYub2JVdubWbmtkRdX1bF/NDQ7HGT8\n8BwmFOUwfngOFcNzmDA8h9EFWYSCmvFS+k7hLtKPhudmMPyYDD55zPADba0dEVbXNbNyWxNr6ppZ\n17CXhet38vS7Ww/0CQcDlBdmMWF4DuMKcxg9LIvyYdmMGZbF8Nx0He1LjxTuIgMsMxyMnYAtyz+o\nvbmtk3UNLdTU72VtfTPr6veyclsTC1bUEYl+eOFDZlqQ0QVZjB6WxZiCLMYUZjOmIIvSoZmMys/U\nE6sEULiLDBq5GWmHDf3OSJStu1vZtGsfm3a2sGnnPjbt3MfGHS28uqaB9q7oQf2HZYcZlZ/JyLwM\nRuVnUpIfC/1R+RmU5GdSmJOu6RZSgMJdZJBLCwYoL8ymvDAbKDpoXTTqqG9uZ9POFrbuaWXbnla2\nNbaxbU8rG3e28EbNDlo6Igd9TShgFOWmMzw3naLcjAPvhw9Jjw0j5aZTFH+ladw/aSncRZJYIGCM\nyMtgRN7h58NxztHU1hUL/W7hX9/UTn1zG7W79/Hu5t3sbOk47NcXZIcpzAkzNCvMsP3/ZocZmh2m\nIP7qvk5DQoOHwl3Ex8yMvMw08jLTmDRyyEf264xE2bG3PR767TQ0x8K/vrmdnXvb2d3SyeoPmtm9\nr5Pd+zr4qHsfs8NBhmaHyc9KY0hG/JUZIi9z//tYLUMyQwcvZ6SRkRbQieIEUriLCGnBACPzMhmZ\nl9lj30jU0djaya6WjgOv3fs6DlpubO2kqbWT9Tv20tTaRWNrJ62dkY/9vmlBIzcjjez0INnhENnp\nsVdOepCscIic9BDZB70PkR0OdusXIiscJCMtSGY4SEYokNKXk/Yq3M1sDnAvEAQecs791yHr04FH\ngE8AO4HLnXMbE1uqiAwGwYAdGJI5Eh1dUZrbOmlq6zoQ/k1tnQfCv6mtk+a2Tva1R9jb3kVLR6x9\n+55WWtq74m2Rg64c6kla0MgIBckIB8lIC5CZFgv//a/MtED83+5tsb7poQBpoQDhYIBwKLYcDgUI\nB4OEQwHSgvZhe7yte3s46O1fIj2Gu5kFgfuAs4Ba4G0zm+ecW9mt2/XAbufcBDO7AvgucHl/FCwi\nySkcCjAsJ51hOX1/gLlzjvauKC3tXezriP8SiId+S/x9W1eUto4IbZ0RWuOvts4obZ0ftrV1Rmhs\n7aSuMUJbV4TWeP+2zigdkWjPhfR2m+O/GA4N/VvOnMgFU0cl7HMOpzdH7icBNc659QBm9jhwIdA9\n3C8EvhN//yTwf2ZmzqtZyUTEl8zswBH2sH76jEjU0dYZob0rSsf+V+SQf7uidEQidHS5g9u6InRE\nonRG3CFfHznwvjPqyM/q/3mGehPuJcCWbsu1wMkf1cc512VmjcAwYEf3TmZ2A3ADwOjRo/tYsohI\n/wkGLD6O73UlR2dAzzY45x50zlU556qKiop6/gIREemT3oT7VqCs23JpvO2wfcwsBOQRO7EqIiIe\n6E24vw1UmNlYMwsDVwDzDukzD7gm/v4S4I8abxcR8U6PY+7xMfSbgAXELoV82Dm3wszuAKqdc/OA\nnwC/MLMaYBexXwAiIuKRXl3n7px7Hnj+kLbbu71vAy5NbGkiItJXqXv7loiIjyncRUR8SOEuIuJD\n5tVFLWbWAGzq45cXcsgNUilA25watM2p4Wi2eYxzrscbhTwL96NhZtXOuSqv6xhI2ubUoG1ODQOx\nzRqWERHxIYW7iIgPJWu4P+h1AR7QNqcGbXNq6PdtTsoxdxER+XjJeuQuIiIfI+nC3czmmNlqM6sx\ns1u9ridRzKzMzF4xs5VmtsLMvhJvLzCzF81sbfzfofF2M7Pvx38Oy8xsurdb0DdmFjSzd81sfnx5\nrJm9Fd+uX8cnq8PM0uPLNfH15V7W3Vdmlm9mT5rZ+2a2ysxmpMA+/mr8v+nlZvaYmWX4cT+b2cNm\nVm9my7u1HfG+NbNr4v3Xmtk1h/us3kiqcO/2yL9zgUrgSjOr9LaqhOkCvu6cqwROAb4U37ZbgZed\ncxXAy/FliP0MKuKvG4D7B77khPgKsKrb8neBe5xzE4DdxB7hCN0e5QjcE++XjO4F/uCcOxaYSmzb\nfbuPzawEuBmocs5NJjb54P5HcfptP/8MmHNI2xHtWzMrAP6V2AORTgL+df8vhCPmnEuaFzADWNBt\n+TbgNq/r6qdt/S2x59auBkbG20YCq+PvHwCu7Nb/QL9keRF7NsDLwOnAfMCI3dgROnR/E5uVdEb8\nfSjez7zehiPc3jxgw6F1+3wf739KW0F8v80HzvHrfgbKgeV93bfAlcAD3doP6nckr6Q6cufwj/wr\n8aiWfhP/U/QE4C2g2Dm3Pb7qA6A4/t4PP4vvAd8E9j+ReBiwxznXFV/uvk0HPcoR2P8ox2QyFmgA\nfhofinrIzLLx8T52zm0F/gfYDGwntt8W4+/93N2R7tuE7fNkC3ffM7Mc4CngFudcU/d1Lvar3BeX\nN5nZ+UC9c26x17UMoBAwHbjfOXcC0MKHf6YD/trHAPEhhQuJ/WIbBWTz10MXKWGg922yhXtvHvmX\ntMwsjViwP+qcezreXGdmI+PrRwL18fZk/1nMBOaa2UbgcWJDM/cC+fFHNcLB2+SHRznWArXOubfi\ny08SC3u/7mOAM4ENzrkG51wn8DSxfe/n/dzdke7bhO3zZAv33jzyLymZmRF7otUq59zd3VZ1f4Th\nNcTG4ve3Xx0/634K0Njtz7/axVoVAAABEElEQVRBzzl3m3Ou1DlXTmw//tE5dxXwCrFHNcJfb29S\nP8rROfcBsMXMjok3nQGsxKf7OG4zcIqZZcX/G9+/zb7dz4c40n27ADjbzIbG/+o5O9525Lw+AdGH\nExbnAWuAdcA/e11PArfrNGJ/si0DlsRf5xEbb3wZWAu8BBTE+xuxK4fWAe8RuxrB8+3o47Z/Epgf\nfz8OWATUAL8B0uPtGfHlmvj6cV7X3cdtnQZUx/fzs8BQv+9j4N+A94HlwC+AdD/uZ+AxYucVOon9\nlXZ9X/YtcF18+2uAz/a1Ht2hKiLiQ8k2LCMiIr2gcBcR8SGFu4iIDyncRUR8SOEuIuJDCncRER9S\nuIuI+JDCXUTEh/4f95m5BOAGZZIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2bEwIRskgFVq"
      },
      "source": [
        "## 2.  create_model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jjosMYBJYk4J",
        "colab": {}
      },
      "source": [
        "def create_model():\n",
        "  model = models.Sequential()\n",
        "  \n",
        "#   model.add(Flatten(input_shape=(2,)))\n",
        "  model.add(Dense(16 ,input_shape=(env.observation_space.shape)))\n",
        "  model.add(Activation('relu'))\n",
        "  model.add(Dense(16))\n",
        "  model.add(Activation('relu'))\n",
        "  model.add(Dense(16))\n",
        "  model.add(Activation('relu'))\n",
        "  model.add(Dense(ACTION_SPACE_SIZE))\n",
        "  model.add(Activation('linear'))\n",
        "  \n",
        "  print(model.summary())\n",
        "  \n",
        "  model.compile(loss = 'mse', optimizer=Adam(lr=0.001),metrics=['accuracy'])\n",
        "  \n",
        "  return model\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lTRr9PZPZ1-1"
      },
      "source": [
        "## 3. Agent Class"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "w_7sLLVzZ1Tj",
        "colab": {}
      },
      "source": [
        "class DQNAgent:\n",
        "\n",
        "  def __init__(self):\n",
        "    \n",
        "    # Replay memory\n",
        "    self.replay_memory = deque(maxlen=REPLAY_MEMORY_SIZE)\n",
        "    \n",
        "    # Prediction Network (the main Model)\n",
        "    self.model_prediction = create_model()\n",
        "    \n",
        "    # Target Network\n",
        "    self.model_target = create_model()\n",
        "    self.model_target.set_weights(self.model_prediction.get_weights())\n",
        "    \n",
        "   \n",
        "    \n",
        "    # Used to count when to update target network with prediction network's weights\n",
        "    self.target_update_counter = 0\n",
        "    \n",
        "    \n",
        "  \n",
        "  # Adds step's data to a memory replay array\n",
        "  # (current_state, action, reward, next_state, done)\n",
        "  def update_replay_memory(self, transition):\n",
        "    self.replay_memory.append(transition)\n",
        "  \n",
        "  \n",
        "  \n",
        "  # Queries prediction network for Q values given current observation space (environment state)\n",
        "  def get_qs(self, state):\n",
        "    return self.model_prediction.predict(np.array(state).reshape(-1, *state.shape))[0]\n",
        "  \n",
        "  def train(self, terminal_state, step):\n",
        "    \n",
        "    if len(self.replay_memory) < MIN_REPLAY_MEMORY_SIZE:\n",
        "      return\n",
        "    \n",
        "    minibatch = random.sample(self.replay_memory, MINIBATCH_SIZE)\n",
        "    \n",
        "    # Get current states from minibatch, then query NN model_prediction for current Q values\n",
        "    current_states = np.array([transition[0] for transition in minibatch])\n",
        "    current_qs_list = self.model_prediction.predict(current_states)   \n",
        "    \n",
        "    # Get next_states from minibatch, then query NN model_target for target Q values\n",
        "    # When using target network, query it, otherwise main network should be queried\n",
        "    next_states = np.array([transition[3] for transition in minibatch])\n",
        "    next_qs_list = self.model_prediction.predict(next_states)\n",
        "    \n",
        "    target_qs_list = self.model_target.predict(next_states)\n",
        "    \n",
        "    X = []\n",
        "    y = []\n",
        "    \n",
        "    # Now we need to enumerate our batches\n",
        "    for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):\n",
        "\n",
        "        # If not a terminal state, get new q from future states, otherwise set it to 0\n",
        "        # almost like with Q Learning, but we use just part of equation here\n",
        "        if not done:\n",
        "            max_target_a = np.argmax(next_qs_list[index])\n",
        "\n",
        "#             max_target_q = np.max(target_qs_list[index])\n",
        "            max_target_q = target_qs_list[index][max_target_a]\n",
        "            new_q = reward + DISCOUNT * max_target_q\n",
        "        else:\n",
        "            new_q = reward\n",
        "\n",
        "        # Update Q value for given state\n",
        "        current_qs = current_qs_list[index]\n",
        "        current_qs[action] = new_q\n",
        "\n",
        "        # And append to our training data\n",
        "        X.append(current_state)\n",
        "        y.append(current_qs)    \n",
        "    \n",
        "    # Fit on all samples as one batch, log only on terminal state\n",
        "    self.model_prediction.fit(np.array(X), np.array(y), batch_size=MINIBATCH_SIZE, verbose=0, shuffle=False if terminal_state else None)\n",
        "    \n",
        "    # Update target network counter every episode\n",
        "    if terminal_state:\n",
        "        self.target_update_counter +=1\n",
        "    \n",
        "    # If counter reaches set value, update target network with weights of main network\n",
        "    if self.target_update_counter > UPDATE_TARGET_EVERY:\n",
        "      self.model_target.set_weights(self.model_prediction.get_weights())\n",
        "      self.target_update_counter = 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TUDDhaxOadLa",
        "outputId": "459d19dd-2afd-4c16-ccd9-0b5b60d624f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 867
        }
      },
      "source": [
        "agent = DQNAgent()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_8 (Dense)              (None, 16)                80        \n",
            "_________________________________________________________________\n",
            "activation_8 (Activation)    (None, 16)                0         \n",
            "_________________________________________________________________\n",
            "dense_9 (Dense)              (None, 16)                272       \n",
            "_________________________________________________________________\n",
            "activation_9 (Activation)    (None, 16)                0         \n",
            "_________________________________________________________________\n",
            "dense_10 (Dense)             (None, 16)                272       \n",
            "_________________________________________________________________\n",
            "activation_10 (Activation)   (None, 16)                0         \n",
            "_________________________________________________________________\n",
            "dense_11 (Dense)             (None, 2)                 34        \n",
            "_________________________________________________________________\n",
            "activation_11 (Activation)   (None, 2)                 0         \n",
            "=================================================================\n",
            "Total params: 658\n",
            "Trainable params: 658\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n",
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_12 (Dense)             (None, 16)                80        \n",
            "_________________________________________________________________\n",
            "activation_12 (Activation)   (None, 16)                0         \n",
            "_________________________________________________________________\n",
            "dense_13 (Dense)             (None, 16)                272       \n",
            "_________________________________________________________________\n",
            "activation_13 (Activation)   (None, 16)                0         \n",
            "_________________________________________________________________\n",
            "dense_14 (Dense)             (None, 16)                272       \n",
            "_________________________________________________________________\n",
            "activation_14 (Activation)   (None, 16)                0         \n",
            "_________________________________________________________________\n",
            "dense_15 (Dense)             (None, 2)                 34        \n",
            "_________________________________________________________________\n",
            "activation_15 (Activation)   (None, 2)                 0         \n",
            "=================================================================\n",
            "Total params: 658\n",
            "Trainable params: 658\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Bpa2BCspawS3"
      },
      "source": [
        "## 4. Train the agent"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "I7j_aM56tO2_",
        "colab": {}
      },
      "source": [
        "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Xmoxp-PEjN3s",
        "outputId": "8ff7730f-6793-4b16-d486-1f6caaaa0003",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Iterate over episodes\n",
        "for episode in tqdm(range(1, EPISODES + 1), ascii=True, unit='episodes'):\n",
        "\n",
        "\n",
        "    # Restarting episode - reset episode reward and step number\n",
        "    episode_reward = 0\n",
        "    step = 1\n",
        "\n",
        "    # Reset environment and get initial state\n",
        "    current_state = env.reset()\n",
        "\n",
        "    # Reset flag and start iterating until episode ends\n",
        "    done = False\n",
        "    while not done:\n",
        "\n",
        "        # This part stays mostly the same, the change is to query a model for Q values\n",
        "        if np.random.random() > epsilon:\n",
        "            # Get action from Q table\n",
        "            action = np.argmax(agent.get_qs(current_state))\n",
        "        else:\n",
        "            # Get random action\n",
        "            action = np.random.randint(0, ACTION_SPACE_SIZE)\n",
        "\n",
        "        next_state, reward, done, _ = env.step(action)\n",
        "\n",
        "        # Transform new continous state to new discrete state and count reward\n",
        "        episode_reward += reward\n",
        "\n",
        "#         if SHOW_PREVIEW and not episode % AGGREGATE_STATS_EVERY:\n",
        "#             env.render()\n",
        "\n",
        "        # Every step we update replay memory and train main network\n",
        "        agent.update_replay_memory((current_state, action, reward, next_state, done))\n",
        "        agent.train(done, step)\n",
        "\n",
        "        current_state = next_state\n",
        "        step += 1\n",
        "\n",
        "    # Append episode reward to a list and log rewards (every given number of episodes)\n",
        "    ep_rewards.append(episode_reward)\n",
        "    if not episode % AGGREGATE_STATS_EVERY or episode == 1:\n",
        "        average_reward = sum(ep_rewards[-AGGREGATE_STATS_EVERY:])/len(ep_rewards[-AGGREGATE_STATS_EVERY:])\n",
        "        min_reward = min(ep_rewards[-AGGREGATE_STATS_EVERY:])\n",
        "        max_reward = max(ep_rewards[-AGGREGATE_STATS_EVERY:])\n",
        "        \n",
        "        aggr_ep_rewards['ep'].append(episode)\n",
        "        aggr_ep_rewards['avg'].append(average_reward)\n",
        "        aggr_ep_rewards['min'].append(min_reward)\n",
        "        aggr_ep_rewards['max'].append(max_reward)\n",
        "        \n",
        "#         agent.tensorboard.update_stats(reward_avg=average_reward, reward_min=min_reward, reward_max=max_reward, epsilon=epsilon)\n",
        "\n",
        "        # Save model, but only when min reward is greater or equal a set value\n",
        "#         if min_reward >= MIN_REWARD:\n",
        "#             agent.model.save(f'models/{MODEL_NAME}__{max_reward:_>7.2f}max_{average_reward:_>7.2f}avg_{min_reward:_>7.2f}min__{int(time.time())}.model')\n",
        "\n",
        "    # Decay epsilon\n",
        "    if epsilon > MIN_EPSILON:\n",
        "        epsilon *= EPSILON_DECAY\n",
        "        epsilon = max(MIN_EPSILON, epsilon)\n"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|##########| 1000/1000 [45:33<00:00,  3.57s/episodes]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QQ22Ry_q4mQS"
      },
      "source": [
        "## 5. Plot episodes vs rewards"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "G1TtpQaZVJDA",
        "outputId": "3b1c022e-ba9c-4fba-9d5f-8dc26cfff01c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        }
      },
      "source": [
        "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
        "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
        "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
        "plt.legend(loc='upper left')\n",
        "plt.xlabel('Episodes')\n",
        "plt.ylabel('Rewards')"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'Rewards')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXd8FGX6wL/vppIOIZ1AQkLvgQAC\nIk3BAgI2xENQFD3l1DtPD/U8f3qe3p16px5WFBALFrBgh6CIIJHeCZCekErKpm7K7vv7Y3YxQEg2\nye7Obpjvh/ns7jsz7zxLknnmfaqQUqKhoaGhoXEuOrUF0NDQ0NBwTjQFoaGhoaHRLJqC0NDQ0NBo\nFk1BaGhoaGg0i6YgNDQ0NDSaRVMQGhoaGhrNoikIDQ0NDY1m0RSEhoaGhkazaApCQ0NDQ6NZ3NUW\noCN0795dxsTEqC2GhoaGhkuxZ8+e01LKkNaOc2kFERMTw+7du9UWQ0NDQ8OlEEJkWXOcZmLS0NDQ\n0GgWTUFoaGhoaDSLpiA0NDQ0NJrFpX0QzdHQ0EBubi4Gg0FtUeyGt7c3PXr0wMPDQ21RNDQ0OjGd\nTkHk5ubi7+9PTEwMQgi1xbE5UkpKSkrIzc0lNjZWbXE0NDQ6MXYzMQkhooUQPwohjgohjggh7jeP\ndxNCbBJCnDS/djWPCyHEy0KIVCHEQSFEQnuuazAYCA4O7pTKAUAIQXBwcKdeIWloaDgH9vRBNAIP\nSikHAmOBe4UQA4FlwGYpZR9gs/kzwJVAH/O2BHitvRfurMrBQmf/fhoaGs6B3UxMUsp8IN/8vlII\ncQyIAq4FJpkPewfYAvzFPL5GKj1Qk4UQQUKICPM8GkBVfRU1jTUAVNZXsnzfcpUlcn7iu8Yzvdd0\nTalaQXFNMetPrqfR1Ojwa+ecPECZ9KfRqys6ncBdJ3Azb7+915013nQfgNEkaTRJjObtt/emC4xL\nvAyn8ZHVBEUPpJuvp8O/d0eI7xrPjJgZdr2GQ3wQQogYYATwKxDW5KZfAISZ30cBOU1OyzWPnaUg\nhBBLUFYY9OzZ024yOyN51Xk0GBsARUG8efRNlSVybiRKv/XNMZt54pIn8PP0U1ki50VKybKfl7Gz\nYCcCFZSpNCkvtQLpoOsrV1J+R2Tqr4Cw/HN6JDA06DLXVxBCCD9gPfCAlLKi6ZOclFIKIWRb5pNS\nvgm8CTBq1Kg2nevKmKSJBmMDIT4hhPqEoivWcXDhQbXFcmpM0sTKwytZvm85R0qO8NxlzzEoeJDa\nYjklP+b8yM6CnTwy+hHmD5jv0GuvW/c+1x++h8agWNzLM6gbcD36qf/CIHyoazRS12iirtFEfaPp\nzOf6JmOGBiNCgJe7G17uOjzddU1e3c589vbQ4enmhrexgqCNf8Qz9RtMsZMQGT9xMO4uXmy8ju2p\nJdQ3mvDzcufSPt2ZOiCMSf1C6O7n5dD/k5Z4/ac0/vltCr3G2P8B2a4KQgjhgaIc3pdSfmoeLrSY\njoQQEUCRefwUEN3k9B7mMZdk9uzZ5OTkYDAYuP/++zGZTKSlpfHcc88BsHr1anbv3s3y5cv5+9//\nznvvvUdISAjR0dGMHDmSP//5z2fNV2esA8Dbzdvh38VV0Qkddwy5g5FhI3nop4dY8M0CHhz1IPP7\nz9dMTk1oMDbwwu4X6B3Ymxv63eDQa0spKTqplMtxX/w97H0Hry3PElq0H65fBRFDbXvBnF2w7nao\nzIMr/oHuknvhvbkMK/6KVQ88Q02j5JfUEjanFPFDSiHfHi5ACBgeHcTU/qFM6R/GgAh/1X5/3kvO\n4p/fpnDN0Aj+fu1gu1/PbgpCKP+DbwPHpJT/abJrA7AQ+Kf59Ysm40uFEB8CYwB9R/0PT355hKN5\nFR2Z4jwGRgbwxMzWn0JXrlxJt27dqK2tJTExkc2bNzN+/PgzCuKjjz7iscceY9euXaxfv54DBw7Q\n0NBAQkICI0eOPG++ukZFQXi5Oc+TjKswInQE62au46/b/8o/d/6Tnfk7eWr8UwR6BaotmlPwQcoH\nZFdm89q01/DQOTa35kheBWE1J6jxCcXHPwwuexh6jYP1d8Bb02DGMzBqMXT0hmwywY7lsPlJCIiE\n2zdCD/Pf2YgFsO42SP8Rn/hpTBsYxrSBYUg5mCN5FWw+piiL5zee4PmNJ4gM9GbKgFCm9g/jkrhg\nvD3cOv4fYQWf7zvF418cZmr/UP570/Azvhd7Ys8opvHAAmCKEGK/ebsKRTFcLoQ4CUwzfwb4BkgH\nUoEVwD12lM3uvPzyywwbNoyxY8eSk5NDRkYGvXv3Jjk5mZKSElJSUhg/fjzbt2/n2muvxdvbG39/\nf2bOnNnsfHXGOgQCDzctOa49BHkH8b8p/+PPo/7M1tyt3PjljRwoPqC2WKpTaijljQNvMD5qPBOi\nJjj8+hsO5DFIl4VHVJOVQswEuHsbxF4KXz8InywEg779F6kugbU3wabHod+VcNfPvykHgP5XQ5du\nsHfNWacJIRgcFcj90/rwxdIJ7HxsKv++bihDegTy6d5T3LZ6F6OeTuLrg/aPo9l4pIAHPznA2Nhg\nXrklAQ83xxTBsGcU0zYu7O+Z2szxErjXljJY86RvD7Zs2UJSUhI7duzAx8eHSZMmYTAYmDdvHh9/\n/DH9+/dnzpw5bVqm1hnr8HTzRCe06ijtRQjBwkELSQhN4KGtD7Ho20Xcn3A/tw669aL9f311/6vU\nNNbw0KiHHH5tk0ny/f5MHhZ5uEfdePZO3+4w/xP45WXY/BTk7YcbVkHU+avrFsn6BdYthprTcNXz\nkHjH+asRdy8YdjPsfBOqTyvXboZQf29uTIzmxsRo6hqN/JpeyotJJ7j3g70cL+zDA1P7oLPDU/22\nk6dZ+sE+BkcFsmLhKIetWECrxWQX9Ho9Xbt2xcfHh5SUFJKTkwGYM2cOX3zxBWvXrmXevHkAjB8/\nni+//BKDwUBVVRVfffVVs3PWGevwctfMS7ZgSMgQPp75MZOiJ/HCnhdYunkpZYYytcVyOCfLTvLJ\niU+4sd+NxAXFOfz6u7PK8KtMwx0jhA85/wCdDiY8ALd9q0Q5vT0ddrwK0orYFJMRtj4Hq68GD2+4\nIwlG33lhU1XCAjA1wIEPrZLdy92NiX1DWLtkLDeM7MHLm0/y+/f3UF1n2xDhPVml3LlmN71DfHnn\ntkT8vBxb/EJTEHZgxowZNDY2MmDAAJYtW8bYsWMB6Nq1KwMGDCArK4vRo0cDkJiYyKxZsxg6dChX\nXnklQ4YMITDwbNu4SZqoN9Zr/gcbEuAZwH8m/YdHxzxKcn4y1395PXsK96gtlsOQUvLcrufw9fDl\nnmHqWHO/2H+KYR7ZyofwFpzRPcfAXVuhz+Xw/SPw4XyoKb3w8VVF8N5c+OFpGDQHlvwEEcNaFiZ0\nAPRIVMxM1iggM17ubvz7+qE8fs1ANh0t5LrXfiGntMbq81viSJ6eRat2ERbgxZrFownyUSFPQ0rp\nstvIkSPluRw9evS8MWensrJSSilldXW1HDlypNyzZ89Z+2sbauXh4sOy3FB+ZswVv6ezcvT0UXnV\n+qvk0HeGyjcOvCGNJqPaItmdn3J+koNXD5ZrjqxR5fr1jUY5/Mnv5Y//XSTl0xFSGq34PzeZpNzx\nqpRPBkv5wkAps5LPPybtRyn/HS/l30Ol3L1KOcda9rwj5RMBzc9rBVuOF8nBT3wnRzy1USannW7X\nHBZSiyplwlMb5SXPJMmc0uoOzdUcwG5pxT1WW0E4AUuWLGH48OEkJCRw3XXXkZBwdhkqS4irtoKw\nDwOCB/DRNR8xvdd0/rfvf9y96W5O155WWyy70WBq4LldzxETEMO8fvNUkWFb6mnKahoY5p4N4YMV\nc1JrCAFjfw+LN4KbO6y6Erb9V4lQMjbCD/+ANbOhSxDc+QOMXNS26KdBc8DDF/ataf3YZrisbwhf\n3DueIB8PbnnrV9buzG7XPLllNfzurV8RAt67Yww9uvq0ax5b0OmquboiH3zwQYv7LQrC0821SgG4\nEn6efvxr4r8YEzGGZ3c+y/UbruefE//J2Iixaotmcz4+/jGZFZksn7Jctai4L/fnEeitI6giBWLb\nqKSiEhST04Y/QNL/QeY2aKiFrO0wbD5c/Tx4+rZdKC9/GDwXDq+H6c+Cd0Cbp+gd4sdn94znvrX7\neOTTQ6TkV/DXawZaHXVUVGHglrd+pbqukQ+XXELvEHWz/7UVhAtQ16hFMDkCIQTX9b2OD67+gACv\nAO7adBcfpljntHQVyg3lvLr/VcZGjGVij4mqyGBoMPL9kQLm9xWI+qrmHdSt4R0IN7wDV78AGT9D\n3j6Y/RrMea19ysFCwkJoqIEjn7Z+7AUI7OLBykWJ3HlpLO/syGLhyp2UVde3el55TT0L3t5JcWUd\nq28fzcDItisoW6PdcVwAg9GgmZccSN+uffnw6g+ZGDWRf/z6D/6z5z+YzLWCXJ3XDrxGVUMVDyU+\npFo28OZjRVTXG7k2vEQZaI+CAMV8lHgH3LMD7kmG4TYoEdJjFIQMOC8noq246QSPXT2Q528Yxu7M\nMma/up2ThZUXPL6qrpGFK3eSUVLNiltHkdCza4eubys0BeHkOEsEU12jEaPpoil9hY+HD/+d/F9u\n6ncTqw6vYtnWZdQbW38KdGbSy9P56PhHXN/nevp27auaHBsOnCLU34u+MgOEDkIHdmzC4Djo2ss2\nwgmhhLye2gOFRzo83fUje7B2yViq64zMefUXNh8rPO8YQ4ORxat3cTivglfmJzA+vvk8DDXQfBBO\njuWmpFYORL6+ln9+m8IX+/PMcujw8XTDx9OdLp5u+Hi60cVDefXxcsfH/L6Lp7v5ODf8vNyZMiCU\nUH/XqiPlrnPnsTGPEe4bzkt7X6K4tpgXJ7/osiU6nt/9PF3cu3DvCJvmo7aJCkMDPx4v5pYxPdEV\nHobufcGji2ryNMvQebDpCdj7Llz5z9aPb4WRvbry5R/Gc+ea3dyxZjcPT+/P3Zf1RghBfaOJe97f\ny87MUl68aTiXDwxrfUIHoikIldiwYQNHjx5l2bJlLR6nVgSTocHIm1vTeW1LGiYpuW18DIFdPKit\nN1Jj3mobGqmuM1Jbb6S4qo6a0poz+2vrjdQbfzPLeLnrmD+mJ3dfFkdYgOsoCiEEdwy5gwjfCP66\n/a8s/HYhr057lUi/SLVFaxPbT23n51M/8+DIB+nm3U01Ob4/XEB9o4lZwyJh3SHodYlqslwQ32AY\ncA0c/BAuf1LJtO4gEYFd+OSucTy07gD/+i6FlIIKnpkzhIfXH+SHlCKemTOEa4dH2UB426IpCJWY\nNWsWs2bNavU4RysIKSXfHCrgmW+Ocaq8lquHRLDsyv5Ed2t7qF2D0URtg5H8cgNv/ZzOmh1ZvP9r\nNjeNiubuSXFEBTnZk2MLXN37akJ9Qrn/h/u55ZtbeHXqqwwIHqC2WFbRaGrkuV3PEe0f7fBS3uey\n4UAePbv5MDzYBBW57fc/2JsRC+DIZ5DyFQy+ziZTdvF04383j6B/uD/PbzzB1hPFlNU08OhV/Znv\ngNLd7UHzQdiBzMxM+vfvz6JFi+jbty+33HILSUlJjB8/nj59+rBz505Wr17N0qVLAVi0aBH33Xcf\n48aNo3fv3qxbt+7MXI6swXQkT89NbyZz7wd7CejiwYdLxvLKLQntUg4AHm46Arw96Bfuz3M3DOPH\nBydxXUIUa3dmM+m5H3nk00M2yzp1BInhiay5cg3uOncWfbeI7ae2qy2SVaw7sY40fRoPjnxQ1VDp\n4so6tqeeZtawSEThYWXQWRVE78kQ2LPDzupzEUKwdEof3lwwEgk8MK0PSyY6vsyJtXTuFcS3y6Dg\nkG3nDB9ilV0yNTWVTz75hJUrV5KYmMgHH3zAtm3b2LBhA8888wyzZ88+6/j8/Hy2bdtGSkoKs2bN\n4vrrrwfMNZjsvHooqarj+Y0n+GhXNkE+njwzZwg3JUbbvJxwz2Afnp07lHsnx/P6T2l8vCuXT3bn\nMDchinsnx9MruAPhiQ4ivms871/1Pvduvpd7N9/L3y75G3P7zFVbrAuir9Pzyv5XSAxPZErPKarK\n8s2hfEwSZg2PhLRNymCYkyoInQ5G/A62PANlmdA1xqbTXzEonGkDwuxS3M+WaCsIOxEbG8uQIUPQ\n6XQMGjSIqVOnIoRgyJAhZGZmnnf87Nmz0el0DBw4kMJCJdJBSmnXCKYGo4m3t2Uw6fktfLI7h9vG\nx/Ljnycxf0xPu9aa79HVh6dnD+Gnhyfxu7G9+Hx/HlNe+Ik/fbyf9OIqu13XVoT6hLJ6xmrGRIzh\niV+e4JX9ryDbUL/Hkbxx8A30dXoeTnxY9SZJGw7k0T/cn75h/sqDm38E+IWoKlOLDJ8PCNj3vl2m\nd3blAJ19BWGDCIT24uX1201dp9Od+azT6WhsPL/iY9PjLTebemM9Ukq7KIgtx4v4+1dHSSuu5rK+\nITx+zUDiQx2btRkR2IX/mzWIeybF8cbWdN7/NYvP951i5rBIlk6Op0+Yv0PlaQu+Hr4sn7qcp3Y8\nxesHXie/Kp8nxj3h8IY7LZGpz2TtsbXM7TOX/t36qypLTmkNe7LKeHhGP2Wg4JDzmpcsBEVD/FTY\n9x5MWgY6x5XZdhY6t4JwcezhoE4vruLpr4/xQ0oRsd19WbloFJP7har6dBka4M3j1wzk95PiWPFz\nOu/uyGLDgTyuGhLBH6bE0z9c/YzS5vDQefDUuKeI9I3k1QOvUlRTxH8m/Qc/T3XLI1h4Yc8LeLp5\nsnTEUrVF4cuDSpj0zKGR0FgHp49DvxkqS2UFCbfCx7dC6mboe4Xa0jgcu5mYhBArhRBFQojDTcY+\natJdLlMIsd88HiOEqG2y73V7yeVK2LIGU6WhgX98fZTpL25lV0Ypj101gO8fmMiU/mGqmx4sdPfz\n4pErB7DtL1O4Z1IcPx0vZsaLP/PHj/ZTUlWntnjNIoTg98N/z1PjnmJXwS4WfreQwurzk6EcTXJ+\nMltytnDn0Dvp3kX9xKsN+/NI6BmkBDwUp4Cp0flXEAB9rwSf7u0u4OfyWFPytT0bMBFIAA5fYP8L\nwN/M72MudFxLW2cp930hcipy5PHS483ua8v3rKlrlNcu3yZjln0l/7LugCyqMNhKRLtSVl0n//Xt\nMRn/6Ndy2JPfy09250hTW8o3O5jtudvl6PdGy6kfT5UnSk+oJkejsVHO+WKOnL5uujQ0qv+zPl5Q\nIXv95Su5alu6MrD3XaWs9ulUdQWzlu8elfLJblJWFqotic1A7XLfUsqtQLNdPYTyyHojsNZe1+8M\n2CKCyWiS3P/hPg7klvPaLQn887qhhPi7Rl2nIB9PHp7Rn6/vu5S4ED/+/MkBbnnrVzJPV6stWrOM\nixrHO1e+g0mauPXbW3lxz4scKznmcAf2p6mfcrLsJH8c+UfVS7SAsnrQCbh6qDm5sOCQUla7a6y6\ngllLwq3KiufAxXe7UiuK6VKgUEp5sslYrBBinxDiJyHEpSrJ5TRIKW2iIJ7++igbjxbyt2sGMmNw\nhI2kcyx9w/z55K5LeHr2YA7l6pn+4lZe3ZJKg9H5Cuj179af9696nxGhI1h9ZDU3fnUjMz+fyct7\nX+ZE2Qm7K4vK+kqW71tOQmgCV/RS32YupWTDgTzGx3f/7cGk4BCEDbKuB4QzENIPoscopTecNFrN\nXqjlpL6Zs1cP+UBPKWWJEGIk8LkQYpCUsuLcE4UQS4AlAD17Omf2oS2oN3U8guntbRms2p7J7eNj\nuW28izytXQCdTvC7sb24fGAYT3xxhH9/d5wN+/N4du4QRjhJ5UsLEX4RvDrtVcoMZSRlJ/F95ve8\nffhtVhxaQWxgLDNiZjA9ZrrN+kBXN1RzrOQYR0qO8FPuT5QZynh12qtO4Vs6kKsnu7SGpVPilQEp\nFQUx5AZ1BWsrCbfCF/dCdrJzlAf57G7o1hsue9iul3G4ghBCuANzgZGWMSllHVBnfr9HCJEG9AV2\nn3u+lPJN4E2AUaNGdVp1XtfYsQimbw/l8/TXR5kxKJzHrnaNkhDWEBbgzesLRvL9kQKe+OIIc1/7\nhYWXxPDn6f0c3tC9Nbp6d+WGvjdwQ98bOF17ms1Zm/ku8zteP/A6rx14jfigeKbHTGdGzAxiAmOs\nmrOmoYbjZcc5WnKUI6ePcKTkCBn6DCTKn0K4bzj3JdzHoOBBdvxm1rNhfx6ebjqmDwpXBsqzoK7C\nNRzUTRk4W0m83bvGORRE+hYQ9g+7VeMvahqQIqXMtQwIIUKAUimlUQjRG+gDpKsgm9PQkRDXPVll\nPPDRfoZHB/HivOF2TXpTi+mDwhkXF8zz3x/nnR2ZfH+kgL9fO5hpTlYN00L3Lt25qf9N3NT/Jopr\nitmUtYnvM7/nlf2v8Mr+V+jXtR8zYmcwvdd0ogOiAeV34HjpcY6UHDmjDNL16Wd6U3Tv0p3BwYOZ\nETuDQcGDGBg80CkiliwYTZIvD+YxuX8IgV3M+SGWygbhQ9UTrD14+cGQ6+DAR0p+lbeKFX1NJqgq\nAr9Qu1/KbgpCCLEWmAR0F0LkAk9IKd8G5nG+c3oi8JQQogEwAXdLKZt1cF8s1BnrcNe549bG5JzM\n09XcuWY34YHevHXrKLw9Om9yj7+3B09eO5hrR0TxyPpD3LFmN1cPieCJmQMJdeKKsSE+IcwfMJ/5\nA+ZTUF3ApqxNfJf5HS/tfYmX9r7EgG4DkEhSy1JplEpSZTfvbgwKHsTUnlMZFDyIQd0HEepj/xtE\nR/g1vYTiyjpmDWtSpbTgkLkHhAuuakfcCntWKy1JR92unhy1ZSCNrq0gpJQ3X2B8UTNj64H19pLF\nFakz1uHt3rabXGl1PYtW7URKyerbRhPsp34EiyNI6NmVL/8wgRU/p/PS5pNsPVnMI1cOYF5itNOX\nMwj3DWfBwAUsGLiAvKo8NmZu5IecH+ji3oXbBt92RhmE+ThPvoq1bDiQh6+nG1MHNLmRFRyC4D7g\n2b4CkKoSlQChgxQzk5oKorpIeXVlBaHRfqS5BpOPh/V/RIYGI3e8s4s8vYG1d44htrvzF76zJZ7u\nOu6dHM9VQyJ49NNDPPrZIT7bl8uzc4c6vIRIe4n0i2TR4EUsGrxIbVE6TH2jiW8PF3DFoPCzV7EF\nh5SIIFdECMVZ/d1f1C0VUmVOxPS1v4JwkTgz18Kact87d+7kkksuYcSIEYwbN47jx48D8N///pfb\nbr8NkzSRdiyNwYMHU1PTcklsk0nyx4/2sy+nnJduGs7IXuo1hFGb2O6+fHDnGP59/VBOFFbxu7d+\npa7RqLZYFx1bTxSjr21QGgNZqCkFfY7rOaibMvRGcPNUQl7VoqpYedVWEB3jXzv/RUppik3n7N+t\nP38Z/ZdWj2ut3PeaNWv4+eefcXd3JykpiUcffZT169dz//33c+lll5L0dRLvvPwOb7zxBj4+La8k\nnvnmGN8eLuCvVw/gyiGumetgS4QQ3DgqmrAAbxau3MlXB/K5bmQPtcW6qPjiQB5dfTyY0KeJ09zZ\ne0BYg083GDDzt25zarRLtawgNAXhuljKfQPNlvvW6/UsXLiQkydPIoSgoaEBUKq9vvTGS0xMnMhd\nd93F+PHjW7zO6u0ZvLUtg0XjYlg8wbVzHWzNxD7d6RPqx9vbMpibEOVyNnxXpaa+kaSjhcxNiMLD\nrYmR4kwEkwsrCFDMTIfXw7GvYKgK+RzVRcoqxjvI7pfq1ArCmid9e9Faue/HH3+cyZMn89lnn5GZ\nmcmkSZPOHH/8+HF8/XwpyC9o8RobjxTw5FdHuXxgGI9fM1C7AZ6DEILFE2JZ9ukhdqSXMC7OeUJA\nOzObjhZS22A827wEioLwC3fIk69diZkIQb2UAn5qKIiqYsX/4IC/d80HoRJ6vZ6oKCX8b/Xq1WeN\nP/7w43z0zUeUlJSc1X60KfWNJu77cB9DewTx8rwRnTLXwRbMHhFFN19PVm7LUFuUi4YvD+QREehN\nYsw5vjBX6AFhDTqd0rM6YyuUqpCuVVXosEZLmoJQiYcffphHHnmEESNGnNVA6IEHHmDe7fMY0G8A\nb7/9NsuWLaOoqOisc+sajZRU1RHi78XbC0fRxbPz5jp0FG8PN343thdJx4pcoludq1NeU89PJ4qZ\nOSzy7BDjxjqlzHdnUBCgdJsTOqWZkKOpLnJIBBN0chOTWsTExHD48Jk2GGetEJruO3HixJnxp59+\nGoA333qTE2Un8HL3IiI6gtTU1LPmbjSayDytRDWtvm003S+SXIeOsGBsL17fksaq7Zn8ffZgtcXp\n1Hx7uIAGozzfvORKPSCsITAK4qfB/g9g0qPg5sBbaVUxRAx3yKW0FYST0VKJDZNJklVSQ73RRDc/\nL+JCXCO+X21C/L24dngk6/bkUl5Tr7Y4nZoN+/Po3d2XQZHndAF01RIbLTH0JqjMh4KDjrumyQTV\nxQ7z42gKwskwGA1A8woiT19LdX0j0V274OWu/ejawuJLY6ltMPLBzmy1Rem0FFYYSM4oYeawyPMD\nJiw9ILp1okg7y3epbDmYxKbUliplNhxkYuqUdxlHN2ixJfXGetx0brjrzl+yVhkaCezi8VvhMw2r\n6R8ewIT47rzzSyb1jc7XR6Iz8OWBPKSEWcMjz995pgdEJ/KX+ZtzjirzHXdNB+ZAQCdUEN7e3pSU\nlLiskjAYDc2uHqSUNBglnm6CkpISvL2dtxids7L40lgKK+r45pAD/6AvIr48kMfgqIDzTZ+WHhCd\nxf9gwTcUEL/dtB1BlePqMEEndFL36NGD3NxciouL1RalXRRUF9DFvQsGL8NZ40aTpEBvwODjQfdA\nP3r00DKD28plfUKIC/HlrW3pXDu8GTOIRrvZn1POgVw9j17V//ydrtoDojXc3MG3u2NNTNWWMhuO\nKWvf6RSEh4cHsbGuaecsqinixk9u5JHRjzB/wPyz9u3PKefOd7fz1q2jSIx1zp4Hzo5OJ1g8oTeP\nfnaIXzNKGds7WG2ROgVH8ypYtGonUUFduC6hmQeXzuigtuAX7uAVhKVQn5YHcdGRVp4GQHxQ/Hn7\n8strAQgP1ExLHWFuQhRdfTx4W0ucswkpBRXc8lYyPh5urL1zbPMl5l25B0Rr+Ic72AdhKbPhmIZF\nmoJwItL1SlZm76De5+3L1yvYDOTgAAAgAElEQVQmpwhNQXSI3xLnCsk8Xa22OC7NicJKblnxK17u\nbqxdMpaewRcoKllw2HV7QLSGfxhUOtgH4RfmkDIboCkIpyKtPI1Ar0CCvc83fRRUGPB019HN11MF\nyToXC8b2wl0nWLVdW0W0l5OFlcxfkYybTrB2yVh6BbfQf6QzOqgt+IUrmc0mB5WUry5ymHkJ7Kgg\nhBArhRBFQojDTcb+TwhxSgix37xd1WTfI0KIVCHEcSHEdHvJ5cyklacRFxjXrPM0r7yWiEBvzbFq\nA0IDvJk1LIqPd+eir2lQWxyXI7WoiptX/IoQinJosTlVbRnoszuvgvAPB2mC6tOOuV6V45LkwL4r\niNXAjGbG/yulHG7evgEQQgxE6VU9yHzOq0KIThQw3TpSStL0ac2alwAK9AbCnbjPsquxeIKSOLd2\nl5Y41xbSi6uYvyIZgLV3jmk9m7/A0gOik5Y48Q9XXh3lh6gq7BwKQkq5FSi18vBrgQ+llHVSygwg\nFRhtL9mckVJDKfo6PXGBcc3uz9cbiAxSoTlJJ2VgZADj4oJZvT2TBqOWOGcNmaeruXlFMkaTZO2d\nY4gP9W/9pM4cwQSKiQkcE8lkMkLNaYdlUYM6PoilQoiDZhNUV/NYFJDT5Jhc89hFQ0sOaqNJUlhh\n0CKYbMziCbEUVBi0xDkryCpRlEODUfLBnWPpE2aFcgBzD4gw1+8BcSH8zSHnjsiFqClVzFkOyoEA\nxyuI14A4YDiQD7zQ1gmEEEuEELuFELtdNRmuOSwhrs2tIEqq6mg0SS2CycZM7hdK7+6+vL0tw2Uz\n7x1BTmkNN7+ZjKHByPt3jKFfuJXKATq3gxp+u1k7YgVxpsxGJ3BSN4eUslBKaZRSmoAV/GZGOgVE\nNzm0h3msuTnelFKOklKOCglx3H+UvUkrT8PPw49Qn/OftH4LcdVMTLZEpxPcNiGWg7l6dmeVqS2O\nU5JTWsO8N5OpaTDy3h1jGBAR0PpJFhrrO1cPiOZw94Iu3Rzjg6g2l9norCYmIUREk49zAEuE0wZg\nnhDCSwgRC/QBdjpSNrVJ16fTO6h3s1FKWg6E/bguIYogHw/e+lmFzmBOzqnyWm5ekUyloYH3Fo9h\nUGQbk7OKU8DU0LkVBJiT5RyxgrDUYeoEJiYhxFpgB9BPCJErhFgM/FsIcUgIcRCYDPwRQEp5BPgY\nOAp8B9wrpXRQYLFzkFqe2oKDWsuithc+nu7MH92TjUcLySrREucs5JXXcvObyehrG3j/jrEMjmpH\n5m5nd1Bb8AuDKgf4IM4oiE5gYpJS3iyljJBSekgpe0gp35ZSLpBSDpFSDpVSzpJS5jc5/h9Syjgp\nZT8p5bf2kssZKTOUUWooJS6oeQVRoDfg6aYjWEuSswsLx8WYE+cy1RbFKSjQG7h5RTJl1fW8u3gM\nQ3q0s6xDwSHw8IFuzYdudxoctYKoLgJ3b/Bqg5mvg2iZ1E6AJYLpQgoiX69EMGlJcvYhLMCbmUMj\n+Xh3DvraiztxrrBCUQ4lVfW8s3g0w6OD2j9ZZ+wB0Rz+4coKwmTncOkqcy9qB94HNAXhBLQUwQSK\niUkzL9mX2yfEUlNv5KOLOHEup7SGm1ckU1Rh4J3bE0no2bX1ky5EZ+0B0Rx+4Uq/7Vpr077aSVWR\nQ81LoCkIpyBdn46Puw/hvuHN7s/XG4jUFIRdGRwVyNje3Vi9PZPGiyxxLuN0NQ99coDJz2+hQG9g\n9e2jGdmrW8cmLc+GOv3FoSAclQtRXexQBzVoCsIpSCtPo3dg8xFMpjNJclqIq71ZPKE3eXoD3x52\nYAMYFTlRWMn9H+5j6gtb2HAgj9+N7cXmBy8jMaaDygEuHgc1NMmmtvPvTVWhQwv1QSdsGOSKpJen\nMzZybLP7TlfX0WDUkuQcwdT+ocQE+/DWtgyuGRrRaX0+h0/pWf5DKt8dKcDH0407L+3NHZf2JsS/\nmV4O7eVMD4iBtpvTWTlTj8mOCsJkhJoSh2ekawpCZSrqKyiqLWoxggm0HAhHoNMJbp8Qy9++OMLe\n7LKOm1mcjD1ZZbzyYyo/pBTh7+3OfVPiuW18LF3tER1XcAiC4ztnD4hzcYSCqD7t8DIboCkI1Ukv\nN0cwtVCkD7Qsakdx/cgevLDxBG/9nNEpFISUkuT0Upb/eJLtqSV09fHgoen9WHBJLwK8Pex34YJD\nEJ1ov/mdCY8u4BVo33IbZ7KoNRPTRUVLRfpAazXqaHw83bl5dE/e3JpGTmkN0d1c8wlYSsnWk6dZ\n/sNJdmWW0d3Pi8euGsD8MT3x9bLzn72lB0Ti7fa9jjPhH2bfFYQKWdSgKQjVSS1PxdvNm0jfyGb3\n51cY8HATWpKcA1k4rhdv/ZzOqu2Z/G2ma9nQTSZJ0rFClv+YysFcPZGB3jx17SBuHBWNt4eD8hHO\n9IC4CCKYLPiHO0hBaD6Ii4r08nRiA2Nxu0AyUYE5SU6n65wOU2ckIrALVw+N4KNd2VwzLKJj+QAO\nZtmnB/l4dy49u/nwr+uGMGdEDzzdHRyseDFFMFnwC4ecZPvNr5KJSQtzVZmWusiB4oOICLiI/A8m\nE+hzIXMb7H0XfnwGsn91uBh/urwvXX09uemNHbyXnOUS5cBzSmtYtyeX+WN68sODl3FTYk/HKwfo\n/D0gmsM/TCm3Ya/fk6oicO8CXm0otW4DtBWEilTVV1FQXXBBBzUoWdQjol3nCdYqGmqhLAvKMqA0\nA8oylfdlmcq4se7s47f9F65fCQNmOkzEXsG+fPWHCdz/4X7++vlh9ueU8/TswY4z07SDNTsyEUKw\ndHI87m4qPvtdLBnUTfELV35vDeXQxQ5/r5YsageHXmsKQkUy9BnAhWswmUySQn0dEUNc2EGtz4VD\n66D4+G9K4Nza+Z5+0DUWQvpB3xnQNQa6xSpjXv6wdh58vBBmvwbDbnKY6EE+nqxclMhLSSd4+YdU\nUgoqeO2WkU7puK6qa+TDXTlcNSRC3da0lh4QfaapJ4MaNA11tYeCqC5yaB8IC5qCUJE0vbkG0wUU\nRGlNPfVGExEBLqYgTEZI3Qx7VsGJ75T4bf8I5YYfN0VRAF1jzUogBnyCW34yWvA5fHgzfLYE6ish\n8Q5HfRPcdII/XdGPoT2C+OPH+5m5fBsvzxvBxL7O1axq/Z5cKg2N3D4+Rl1BTh+/OHpAnEtTBRE6\nwPbzVxUpfzMORlMQKpJeno6nzpMov+bbb+eXKzkQLlNmo7IQ9r0Le95Rwhx9Q2D8AzByoaII2ouX\nH8z/BD5ZBF8/CHVVMOEBW0ltFdMGhrFh6QTufncPC1ft5M9X9OP3l8U5RfCAySRZtT2DET2DGKG2\nQ/1idFBDk3IbdsqFqCqC6NGtH2djNAWhImn6NGICY3DXNf9jsDQKigxy4hWElJCxFXavhJSvlKqW\nsRPhiqeg39XgbqPwXA9vuOld+OwuSHoC6iphyl8dapON7e7LZ/eOY9n6Qzz3/XH255Tzwo3D7Jtw\nZgU/Hi8is6SGB6/op6ocwMXTA+Jc7Fmwz9hoLrPh2BwI0BSEqqSVpzG0+4WftAoqLCsIJ1QQNaWw\n/33YvQpK0xS765i7YeQi6N7HPtd084C5K8DTF35+HuqrYPqzoHOcQ9bH052X5g1neHQQ//jmGLOX\nb+f1BSPpG+bY6JKmrNyeQUSgNzMGN18N2KFcLD0gzsXLX/Gl2UNB1JwGpMNDXMG+LUdXCiGKhBCH\nm4w9J4RIEUIcFEJ8JoQIMo/HCCFqhRD7zdvr9pLLWahpqCGvKq/FENe8ciVJrruvDYuodQQpITsZ\nPl0CL/SHjX9VfmnnvAl/SoHp/7CfcrCgc4OZL8PYe+HX12HDHxSfhwMRQqnZ9MEdY6gwNDL7le18\ndTDPoTJYSCmoYHtqCbdeEoOHmpFLYO4BcRDCBqsrh1rYq/WoSklyYN8VxGpgObCmydgm4BEpZaMQ\n4l/AI8BfzPvSpJTD7SiPU5FZkYlEXtBBDVCgryUswAmS5BoMim9h90ooOqq0PEy4FUbdpjwtOhoh\nFGXkHQBbnlVWEnNX2M6cZSVjegfz9X0T+P17e1j6wT72Z5ez7Mr+Dg0xXbUtE28PHTePjnbYNS+I\nPgcMF0kPiOawV+tRlcpsgB0VhJRyqxAi5pyxjU0+JgPX2+v6zk5rXeTAnCSntnmpJA0+WaiYDiJH\nwKz/weDrFDOPmggBk5Ypy/qNj0F9teKj8HCsQz8swJsPl1zC018f5a1tGRw6pWf5/ATbls6+ACVV\ndXy2/xQ3jOxBkI8TlGK5WB3UFvzCIH+/7edVKYsa1M2kvh34tsnnWCHEPiHET0KISy90khBiiRBi\ntxBid3Fxsf2ltBNp5Wm4C3eiAy785FegdqOgQ+vgjYlKLsO8tbBki7JyUFs5NGXcUrjmRUhNgveu\nV5zXDsbTXcdT1w7mPzcOY39OOTP/t4292WV2v+77v2ZT32jiNrVDWy0UHAIEhLlW/Sqb4R+h+CBs\nnU2t4gpCFQUhhHgMaATeNw/lAz2llCOAPwEfCCECmjtXSvmmlHKUlHJUSIhzxaK3hTR9Gr0CeuGh\naz4CRkqpXqvR+hrFtr9+sWJPvnsb9L/K8XJYy6jb4Lq3IHsHrLlWcaCrwNyEHnx6zzg83AU3vbGD\nX1JP2+1adY1G3k3O4rK+IcSHqucgP4szPSCc6AHCkfiHQUON7R9SqoqUyDAvP9vOawUOVxBCiEXA\nNcAt0lzgRkpZJ6UsMb/fA6QBfR0tmyNJL09v0UFdWl1PfaPJ8RFMRSmwYgrsXQMT/gSLvoLAHo6V\noT0MuR5uek+pJLr6avvYgq1gUGQgXy29lOhuPjy07iCVhga7XOfrg/kUV9Zx+wTHJ09dkIKDF6//\nAeyXC1FdpIp5CaxUEEKIOCGEl/n9JCHEfZYIpLYghJgBPAzMklLWNBkPEUK4md/3BvoA6W2d31Uw\nNBrIrcpt0UGdr0YnuX3vw4rJSnP0362HaU8ooaWuQv+r4JaPlXIeq66E8hxVxAj08eD5G4aRr6/l\nmW+O2Xx+KSVvb8sgPtSPiX2623z+dlFbDuXZF7eCsFcuRFWRKuYlsH4FsR4wCiHigTeBaOCDlk4Q\nQqwFdgD9hBC5QojFKFFN/sCmc8JZJwIHhRD7gXXA3VJKdewEDiCrIguTNFmpIBzgg6irgk/vgi/u\ngaiRikkp3kVr6fSepJTmqD4NK2coTnYVSOjZlTsn9mbtzhy2nrCtr2xXZhlH8iq4bXyM8/TNPrVb\neb1YHdSg+CDATgpCncq41ioIk5SyEZgD/E9K+RAQ0dIJUsqbpZQRUkoPKWUPKeXbUsp4KWW0lHK4\nebvbfOx6KeUg81iClPLLjn0t58aaCKYCcxa13VcQBYfhzUlw8COY9Ajc+gUEtPijdX56jlFMY40G\nZSVx+qQqYvxxWl/iQ/34y/qDVNjQ1LRyWwZBPh7MHeFEpr8jnysRZTHj1ZZEPSxP+bbOhXB2ExPQ\nIIS4GVgIfGUecyHbg3ORpk/DTbjRK6DXBY/J1xtw1wmC/ewULimlktewYgrUVcDCDUrYaGfJgI0Y\nCrd9oxQKXH2NKisJbw83nr9hGIUVBv7xlW1MTTmlNWw8WsD80T3p4ukkP6vGeji2Afpf7fAwY6fC\nOxDcvW27gjA2qFZmA6xXELcBlwD/kFJmCCFigXftJ1bnJr08nWj/aDzdLhy7nq83EBbgjZs9kuQM\nFbDudvjqj8oT393blfpJnY2QfrDwS6W66OproNTxbq3h0UHcdVkcH+3O4cfjRR2e751fMtEJwYJL\nLvxw4XDSflAS5AZftGlNCkKYs6lt6KSuNkfC+TnxCkJKeVRKeZ+Ucq35c4aU8l/2Fa3zkqZPa9H/\nAEqhPruYl/L2KbkNR7+AqU/ALetV++VzCKEDFCXRaIDVM5UGRQ7mgWl96Bvmx7L1B9HXtt/UVFXX\nyEfmng8O8U1Zy+H1Si2u3pPUlkR9LLkQtqJavRwIaEVBCCEOmesmNbs5SsjORIOxgeyKbHoHtlzt\n0tKL2mZICb++AW9fAcZ6xfxy6Z8cWuhONcIGKSa0hmp4Z6bStc6BeLkrpqbTVfX8/auj7Z5n3e4c\nKusanSu0tb4Gjn8DA2Y5vNSJU+IfZlsFYUmSU6FZELS+grgGmAl8Z95uMW/fAt/YV7TOSWZFJkZp\nbHEFYUmSs+kKYsdy+PZhpWHP3dug51jbze0KhA9RopvqKuCdaxweAju0RxC/vyyOdXty+SGl7SYI\nk0my6pdMEnoGMTy6zRHm9uPkRqUW1uDr1JbEOfALt62J6UwWtROamKSUWVLKLOByKeXDUspD5u0v\nwBWOEbFz0VoXOYCymgbqGk22NSOk/wShA+HmD8Gnm+3mdSUihytKolavKAn9KYde/g9T4+kf7s+y\n9YfQ17TN1PRDShFZJTWtrx42Pq6ELDuKw+uVp9uYCY67pjPjH6Y8hNRX22a+audeQVgQQojxTT6M\na8O5Gk1IL09HIIgJiLngMfn2CHEtPq6YWpwlbl4tohJgwWdKOY53roEKx5XptpiaSqrrefLLI206\nd+X2DCIDvZkxqIWeD1Iq4coHP4TTqR2U1goMFcoKYtCczhP91lFsnQtRVQQevqqU2QDrb/K3A68K\nITKFEJnAq+YxjTaSVp5GD/8eeLtf+OZfoLdxo6C6KqUFaIgTdBxzBnqMhN99ClXFik/CHk1eLsDg\nqEDunRzPp/tOsemodaaIY/kV/JJWwq3jYlouJV6W8Zt5Y/dKG0jbCse/VZz/mnnpN87kQtjIzFRV\npGoQSasKQgihA+KllMOAYcAwc0LbXrtL1wlJ16e3mCAHkGdWEJFBNjIxnT6hvIb0t818nYHoRPjd\nOqjINysJx9VuWjo5ngERATz62SHKqutbPX7V9gy6eLgxL7GVng/Zycpr2BDY/57iQLYnh9dDYDT0\nSLTvdVwJf/MKz2YriELVIpjACgUhpTSh1E9CSqmXUurtLlUnpcHYQGZFZotF+kDJonbTCbrbKkmu\n+LjyqimIs+k5VlES+lxYM0tZUTgAT3cdz98wlLLqev6vFVPT6ao6Pt+fx3Ujo1rv+ZC9A7yDYMYz\nSl7CkU9tKPU51JRC2mazeUmzNp/B1gX7qotVy6IG601MSUKIPwshooUQ3SybXSXrhOws2EmjqZER\noSNaPC5fbyDM38t2SXLFKeDmCV2dKDzSWeg1DuZ/rIS+rpn1W2KSnRkUGcjSKfF8sT+P7w5f+Gnz\n/WSl58OicVb87LKTFaUXcyl07we73rahxOdw7EswNWrmpXPx6QY6D6jMt818KhbqA+sVxE3AvcBW\nYI95220voTorm7I24ePuwyWRl7R4XH65gQhbmZdAURDBfcDNnh1mXZjYS2H+R0qm9ZprobrEIZe9\nd3I8AyMC+OvnhyhtxtRk6fkwqV8I8aGtOCmrTyumxJ5jlUCExMWQt1dJjLQHh9dBtziIGGaf+V0V\nIWzXetTYALWlqhXqA+szqWOb2Vq2k2ichdFk5MecH5nYYyJebi2bjpROcraMYErRHNSt0fsyuHmt\nUtjvXcc0HfJw0/HCjcPQ1zbwxIbzTU1fHcjndFUdi61JjLP4H3qaHz6GzVOazNhjFVFZABk/K6uH\niz0qrjn8wmxTsK/abPJ0ARMTQojBQogbhRC3WjZ7CtbZ2Fe0j1JDKdN6tVxGW0mSqyUiwEYKor5G\nMZ9o/ofWiZsCN3+g+GzenQ219m8bOiAigPum9OHLA3l8e+g3s4SUkpXbM+gT6seEeCt6PmTvADcv\npW84KIXjhlyvtI219fc4+gUgNfPShbDVCkLFVqMWrLI5CCGeACYBA1EyqK8EtgFr7CZZJyMpOwlP\nnSeXRl2w3TYA5TUNGBpMtjMxlZwEpLaCsJb4aXDT+/DRLbBiKnTv07553DxhxrNWdeO7e1IcG48W\n8tfPDzM6thvBfl7szCjlSF4Fz84dYl3Ph+xkJcfDvcnqdNRipTPggQ9h7O/b9z2a4/B6pRVtqPbQ\n0Sz+4ZC5rePznFEQTm5iAq4HpgIFUsrbUMJdA+0mVSdDSklSVhLjosbh4+HT4rE27ySnRTC1nb5X\nKErCp5vibGzPdmyDkrRmBR5uOp6/YRiVhkb+9oVialq5PYOuPh7MGRHV+gT1NZC///zyKZHDIWqU\nkhOhdPftOOXZkPMrDJ5rm/k6I37hYCiHBkPH5jmTRa2eiclar2WtlNIkhGgUQgQARShd5TSs4EjJ\nEQprCrkv4b5Wjy2oULKobeaDKE4BnTt001xGbaLvFcrWXl6bAGk/wqUPWnV4v3B/7p/Wh+e+P87g\nLWlsPFrIPZPi8PawIkP51B4loqhnM8EPiYvh899D5s+2Kel+5DPldZCmIC6If5Nkua4dKMtuCZV1\ngRXEbnMP6hUoEUx7UdqJtogQYqUQokgIcbjJWDchxCYhxEnza1fzuBBCvCyESDVXi01ox/dxSjZl\nbcJduHNZj8taPTav3JwkZ6s6TMXHlWgTrdKmY4mfoph96qqsPuWuib0Z1iOQf32XgpsQLBgbY92J\n2cmAgOjR5+8bNEfJjbCVs/rweqUtbTctZPqC2CoXoqpY6dLn6dtxmdqJtVFM90gpy6WUrwOXAwvN\npqbWWA3MOGdsGbBZStkH2Gz+DIpfo495WwK8Zo1szo7FvJQYnkigV+tWuQK9ATedIMTfVklyWgST\nKsRNURoVZW23+hR3s6nJ013HzGGR1q8is3cohRi7dD1/n0cXGPE7SPmq49m9p1Mh/4DmnG6NM9nU\nHcyFqFavF7UFqxSEEOJdIcSdQoj+UspMKaVVvSCklFuBc+MFrwXeMb9/B5jdZHyNVEgGgoQQLt4c\nGVLLU8muzG41eslCvt5AqK2S5BrrlNh+zf/geKLHgnsXSN3cptP6hPmz+U+X8ezcIdadYDJCzs6W\ny7ePul0xQe3tYEzJkU8BoaxKNC7MGQXR0RVEkWpVXC1Ya2JaCUQA/xNCpAsh1gsh7m/nNcOklBbV\nWgBYYriigKZF+nPNY2chhFgihNgthNhdXOyY0ggdISkrCYFgSs8pVh1fUFFrO/9DSarSk1lbQTge\nD2+lnWvaD20+Nbqbj3W+B4DCw1Bf2bz/wUJwHPSeDHtWg7GxzfIAipP70Dol8zwgsn1zXCz4dAfh\n1vFcCJUL9YH1JqYfgX8Aj6P4IUYBHY6bk1JKoE3hFVLKN6WUo6SUo0JCnL9VZlJ2EiNCR9C9ixWx\n7ChZ1LbzP6Qor9oKQh3ipihhxuXZ9rvGmQS5VhpAJS6GilNw8vv2XafoKJw+rkUvWYNOp5iGOryC\nULdQH1hvYtoMbEcpuXEcSJRStveuU2gxHZlfLZ3cT3F2ZFQP85jLkl2RzYmyE1ablyyd5Gy2gihK\nAaFrfyy/RseIM68a03603zWyd0BADwhqJaiw75XgHwm73mrfdQ6vV56KB1zbvvMvNvzDO+aDaKxX\nQmVdxMR0EKgHBgNDgcFCiPY+5m4AFprfLwS+aDJ+qzmaaSygb2KKckmSspMAmNpzqlXHV9Q2Uttg\ntGEORIoS3upuI4e3RtsI6a/clNthZrIKKX8r0Ncabu4wcqEiS0la269zeL1SjkRlk4fL0NHWo5Yy\nG67gpJZS/lFKORGYC5QAq4Dy1s4TQqxFCYftJ4TIFUIsBv4JXC6EOAlMM38GJUM7HUhFMWPd08bv\n4nRsztrMwOCBRPpZZ7PNO9NJzoYhrpp5ST2EUFYR6VsUZ7KtKc9SnlKt7S+esFBZBexZ1bbrnNoL\nZZkw+Po2i3jR4h/WsagxJ8iBAOtLbSwFLgVGApkoTuufWztPSnnzBXad90ht9kfca408rkBBdQEH\nTx/kvhGtJ8edOceWneQa66E0DQZc0/G5NNpP3GSleU/efqWTnS2x+B96jbPu+IAI6H817HsfJv9V\ncaRbw+H1SumQ/le3T86LEb9wqDmtVGR182j7+WcK9bnACgLwBv4D9JdSTpNSPimltNO6uXOwOVsJ\nb7TW/wA2LrNRmq6ENmorCHXpPRkQSnMdW5O9A7wCIWSA9eckLlZKSB/93LrjTSYlvDX+cugS1D45\nL0b8O5gs5wR1mMB6E9PzgAewAEAIESKE0FIpW2Bz9mbiAuOIDbT+vylfX4tOQKgtkuTORDBpIa6q\n4hus9Eywhx8iOxl6jmlbR7fYyyA43npndfYOxYylRS+1jY7mQjiJicnaKKYngL8Aj5iHPID37CWU\nq1NqKGVP4R6m9rLOOW1BSZLzbrkxvbUUHweE0ihIQ13ipijJbIYK281ZU6o8BFjrf7AghJI4l7sL\n8q3Idz28Xukr0e/K9sl5sWIJT21vLkR1MXj6K5nwKmLtnWgOMAuoBpBS5gH+9hLK1dmSswWTNHF5\nr8vbdF6BLUNci1OUQmGeLVeP1XAAcVNAGpWCebbi3AZBbWH4fCXLe3cr9ZmMjYopqu8MVesBuSRn\nVhDtVBBVhaqvHsB6BVHfNKlNCKH9trTApqxNRPlF0a9r28w7efpaIoNsWOZb8z84B9FjwMPXtmam\n7B2K4ziyHTUtu3RV6ikd/AQM+gsfl/ET1JRotZfag28oIDqgIIpdSkF8LIR4A6U+0p1AEtDOjJvO\nTWV9Jcn5yUzrOc26Ri9mpJTKCiLABktKY6OSwav5H5wDd0+l77VNFUSy0j3O2kikc0m8HRqq4UAL\nPSsOfwpeAUoTJY224eau9HFot4lJ/UJ90DYn9TpgPdAP+JuU8mV7CuaqbM3dSqOpsU3RSwAVhkZq\n6m2UJFeWCcZ6bQXhTMRNUSLLSjM6PldDLeTta595yULUSIgYrpiZmmsm1FgHx76E/te0Xwld7PiH\ndcxJrXKIK7ShJ7WUcpOU8iEp5Z+BzUKIW+wol8uSlJVESJcQhoYMbdN5lhyICFuYmIqPKa/aCsJ5\nOFN2wwariFN7lVLiHVEQAIl3KL6qrF/O35e6Ger0mnmpI/iFt28F0VinmP6cfQUhhAgQQjwihFgu\nhLjCXAZjKUrG842OESESr2kAABs6SURBVNF1qG2sZdupbUzpOQWdaFsk0m9Z1LZQEOYQ1+6agnAa\nguMhMNo2CiLb3KuruQZBbWHwdeAd2Lyz+vB66NJNKa+h0T78w9vng3CSMhvQ+griXRST0iHgDuBH\n4AZgtpRSq9p1Dr+c+gWD0dDm6CVomkVtAx9E8XEI7Alefh2fS8M2CKFkVWdsbX/JbQvZyUpynE+3\njs3j6QPD5sPRDb8lZgHUV8Pxb2DQ7PZlAWso+IcrN/u2llmx5EC4gImpt5RykZTyDeBmYCAwXUq5\n3/6iuR6bsjcR6BXIyLC2l1TI1xtsmySnmZecj7gpUFeh9JBuL9Y0CGoLo25XzFVNmwmd+A4aajTz\nUkfxC1P6sVS3sW9NleusIBosb6SURiBXSmmwr0iuSYOxgZ9yfmJy9GTcdVaVuDqL/PJaQvy98Oho\nkpzJCKe1CCanJPYypfx6R8xMRUcV30BH/Q8WQvpCzKVKMyHLk+7hT8E/wnbXuFhpby6Ek2RRQ+sK\nYpgQosK8VQJDLe+FEDZMC3V9fi34laqGqnaZlwAKKgy2MS+VZ0GjQYtgckZ8uil5Cx1RENY2CGoL\niXeAPgdOblKcoyc3Km1FdVZ2tdNoHn9zx+S2Kohqs7nPCUxMLT7qSim13xArScpKwtfDlzERY9p1\nfr7eQHyIDXwGxceVV01BOCdxU+Dn56G2vH3F77J3KD0mgnraTqb+VysRN7vfVgr5Ges185ItaG+5\njapipQijE4QX26Doj4bRZOSH7B+YGDURL7e2+xCklOSX19ooxNVSpK9vx+fSsD1xUxS7dMbWtp8r\nJWTtgF6XKE5vW+HmAQm3KiuIHa8qyifKxqXJL0YsCqKtuRBVhU7TmElTEDZgb9FeyurK2pwcZ6Gy\nrpFqWyXJFR9XnjC9Azs+l4bt6TFKKcLWnvLf+hyozLOPb2DkQkXpFB5SVg+2VEAXK+6eSqhwW1cQ\n1cVOYV4CFRSEEKKfEGJ/k61CCPGAEOL/hBCnmoxf5WjZ2ktSVhJebl5MiJrQrvNtG+KqRTA5NW4e\nEDsRUn9oPoO5Jezhf7AQ2AP6mf/kNPOS7fCPaIeT2jnKbIAKCkJKeVxKOVxKORylQ10N8Jl5938t\n+6SU3zhatvZgkiaSspMYFzkOH4/2VU7NK1eS5CI7uoIwmaD4hOZ/cHbiJoM+Wym90Raydyi1kUIH\n2keuaU/C9GcgbLB95r8YaU/r0YtZQZzDVCBNSpmlshzt5sjpIxTVFLU7egls2Gq0IlcpwKatIJyb\neHOfkLZGM2UnK9nT9oou6h4Pl9yrmZdsiV9427rKNRiUMGZNQQAwD1jb5PNSIcRBIcRKIURXtYRq\nC5uyN+Eu3JnYY2K758jXGxACwgI6qCCKzA7q0Da0oNRwPN16Q9eYtimImlIlB8Ie5iUN++EfpigI\nk8m6450oxBVUVBBCCE+UJkSfmIdeA+KA4UA+8MIFzlsihNgthNhdXNzGDEUbI6Vkc9ZmRkeMJtCr\n/U7hAr2BED8bJMmdqcGkRTA5PXFTzGU3Glo/FpTsadCS11wN/wilN3xNiXXHO1EWNai7grgS2Cul\nLASQUhZKKY1SShOwAmi2EpmU8k0p5Sgp5aiQEHVDwU6WnyS7Mrvd0UsW8vS1totg8gvreI0eDfsT\nNwXqq5TWn9aQvQN0Hlr4qavR1lwIywpCUxDcTBPzkhAiosm+OcBhh0vURpKykhAIJkdP7tA8Nms1\nqkUwuQ4xl4JwU8pqW8OZBkHq9ijWaCNnym1Y6YdwokJ9oJKCMLcsvRz4tMnwv4UQh4QQB4HJwB/V\nkK0tJGUnMSJ0BN27dO/QPAV6AxEdDXGVUmsz6kp0CVJyIqzxQzQYIG+v5n9wRdq6grCYmHwv4kQ5\nKWW1lDJYSqlvMrZASjlESjlUSjlLSpmvhmzWklWRxcmykx2KXgKoNDRQWdfYcRNTRR7UV2orCFci\nborSGa6mtOXj8vYp5S80/4PrcWYFYeXtrKpQSXJ1gjIboH4Uk8uSlJUEwNSeUzs0j81CXM+U2NBW\nEC5D3FRAQvqWlo870yCofXW+NFTEo4tyw7fWxFRd5DTmJdAURLvZnL2ZQcGDiPCLaP3gFsg3K4jI\noA6amLQifa5H5Ajl5tGamSk7WekO6BvsGLk0bEtbWo9WFf9mlnICNAXRDgqqCzh0+lCHo5cA8s2t\nRsM7mgNRnAI+weDbMX+IhgNxc1d6RKT9eOGyGyYT5CRr/gdXxj+sbU5qJynUB5qCaBebs5XIk2k9\nbaEgbJQkpzmoXZO4KUoG/OmTze8vPqb0aND8D65LW+oxOVGhPtAURLtIykoiPiiemMCYDs9VoDfQ\n3c8LT/cO/Cik1EJcXZU4c4j0haq7WvwP2grCdfELU0xMrRVnbKhVWtI6SQ4EaAqizRTVFLG3aK9N\nzEsA/9/evUdHWd4JHP/+knBPIBduAYLhZq2iRIQq1XostF5aq73YorWCrcr2HNva6m6rbs9B155T\nu724tdvjSlsKWmq33lpFq7Jqt3WrCCoXEZA7CAkJmAQTEpLAb/94njeMMRcyM5n3nZnf55w5M+8z\nb+Z9nryQ3zz3ffXNiY9gatgPzXVWg0hHReVQPKnrfojdr7hvoEXlqcyVSaaC0W4UWlNt9+c1RGuS\nHFiA6JX1Neu55ulryJVcLplwSVI+s6q+KTn9D2ABIl1Nmg07X4K2Ix98b7fvf7AF9NJX+1yIHvoh\nGoM5EBYg0oqqsmzjMuY9Mw+ABy55gInDJiblsyuTUYOwEUzpbfIcaD0Me1a+P71uj9skyPof0lv7\n3tQ9zIUIAkiEahDd7klt4L2W91j4j4Ws2LWCC8ZdwA/O+0FCC/PFajjSxnvNbZQmPMR1EwwsjNQ/\nLNML5edBTp5rZpoQsypwEDCs/yG9nehyG9bElF42vbuJK5dfyQu7X+Dms27m3tn3Ji04gGteApJT\ngxhxijVDpKsBBW4SXMd+iN0vu+1JR54WTr5McpzochuN0VpmAyxAdEpVeeTtR7j6qatpbmtm8UWL\n+erUryJJ/gMcTJJLSh+EjWBKb5M+DpVrofHA8bTdr0DZTDdfwqSvAfnQP/8EahD7XUtA3oDU5OsE\nWIDo4HDrYW5/6XbufPlOZoyewcOXPcz0UdP75FpJmUXdeMCtNW/9D+lt0mz3HCy70VQH+zdY/0Om\nKBh9An0Q0dlqNGBfTWJsq9vGzX+9mR31O7ix4kZuOP0Gcvtqe0egss4FiJFDE/jG0D6CyWoQaa20\nAgYVueW/T7/CbxCkFiAyxYlsPdoYrWU2wAJEuye3Pcldr9zFoLxBLLpwEeeU9n3HYNWhJobn92dA\nXgJByIa4ZoacXJh4geuHUPUbBOXZBkGZomAU7H29+3Ma9rsvChGS9QGiua2Zu1+9m0e3PMpZo87i\nx+f/mBGDU9NJVJmMfSBqNruOzKFjkpMpE55Jc2DD41C90fU/lFZA/8Fh58okQ1CDUO16MElDjTUx\nRcmuQ7u45a+3sLl2M9effj03VtxIXk7qfiWVdc2ML0nwD0DQQW0jmNJfsOzG23+Bva/BR24INz8m\neQpGu7kuRw65FXw7ajns9nOJWIDI2k7q53Y+x9zlc6k6XMUv5/ySm6bflNLgAG4l16QNcTXpb9g4\nt6z3K/fB0SPW/5BJepoLEexFHaFZ1BBigBCRnX6L0TUistqnFYvIChHZ4p+L+uLaL+97mVv+9xYm\nFU7i4Usf5vxx5/f8Q0nWeKSNQ81tiTUxHX7XVVutgzpzTJp9fDy8TZDLHD3NhQi2Go1YJ3XYNYiP\nq2qFqs7wx7cCz6vqFOB5f5x0Z5eezcJZC1ly0ZKEN/yJVzDENaEaRLDExsgPJyFHJhKC4a4lU2xv\nj0zSUw2ifZmN6EySg/ADREeXA0v966XAZ/viIjmSwxUnX0G/3H598fEnJClbjdoQ18xTfi7kDYST\nPhp2Tkwy9bQ3dUSbmMLspFbgORFR4H5VXQSMUtXgN1gFRKu+lUTBTnJjEmliqtkM/YbA0HFJypUJ\nXf8hcO3TUDg+7JyYZBowFPIGdT0XoiF6y2xAuAHiPFXdKyIjgRUisin2TVVVHzzeR0QWAAsAxo9P\n3/9EQQ0i4UlyI06GnKhVBE1Cxtnch4wj4rce7aoPYr+bKJnXP7X56kFof1lUda9/rgYeBz4C7BeR\nUgD/XN3Jzy1S1RmqOmPEiGhF297YV99MyZD+DOyXyCQ5G8FkTNrobjZ1Y3XkOqghpAAhIkNEpCB4\nDVwIvAk8Acz3p80H/hxG/lKhqr6J0sIE+h+a6+G9fdb/YEy66G49pobqyDUvQXg1iFHASyKyFngV\neEpVnwHuBj4pIluAT/jjjFRZ38zooYn0P7ztnq0GYUx6KBjdzSim6C3UByH1QajqdmBaJ+kHgTmp\nz1HqVdY3M7O8OP4PsBFMxqSX/FFutnRLoxuMECuCC/VB9Ia5ZoXDLW3UN7UmPsQ1byAUnpS8jBlj\n+k77UNcOHdUtjdDSYE1Mxqlq3wciwUlyw6e4VUCNMdHXVYCI4FajAQsQITi+k1wifRCbrP/BmHSS\n7wNEx+U22gOENTEZkrDMxpH3oH6PBQhj0klXy220z6K2JiaDG+IKCSyzccBGMBmTdgYVQW5/q0GY\n7lXWN1OcyCS5YJE+CxDGpA8R18zUVR9EBBdntAARAjcHIsERTLn9oag8aXkyxqRAZ8ttNFbDoGII\ncfHQrliACEFlfXPiI5hKpkBuVm8IaEz6yR/1weU2GqK5zAZYgAhFVX1T4nMgbIKcMemnoIsmpojt\nAxGwAJFiTS1HqT3cGv9Oci2HoXaX9T8Yk44KRkNzHbQ2HU9rrI7cPhABCxApVnUowSGuB7cAajUI\nY9JR+1yImGYma2IygcpEh7jaCCZj0lfHuRBHGqD1sDUxGaeyLqhBxNnEVLMJcvKgeGISc2WMSYmg\nphDMhWiM7hwIsACRcgk3MVVvguJJkdt5yhhzAgpK3XPQUd0Qzb2oAxYgUqyyvomiwf0SmCRnI5iM\nSVuDS1wLQMcAYU1MBlwT0+h4m5dam6F2B4z8cHIzZYxJjZwcV1sIOqmticnEqqxvZkzcI5i2gh6z\nGoQx6Sx2NnVDNSAwOHrLbEAIAUJEykTkRRF5S0Q2iMhNPv0OEdkrImv841OpzlsqVB1qTmAEU7CL\nnI1gMiZtFZS+P0AMLonsqghh5KoNuEVVXxeRAuA1EVnh37tHVX8SQp76RNvRY2ypbmDtnjrWvlPH\n2j31vNvYwpjCeEcwbQbJgZLJyc2oMSZ18kfBnpXudUT3og6kPECoaiVQ6V+/JyIbgbGpzkeyqSrv\n1Db5QOCCwfq99TS1HgVg6MA8ppUV8s3Zk5k7syy+i9RscsNb8wYkMefGmJQqGA2HD0Jbi59FHc0O\naginBtFORMqBM4GVwLnAN0RkHrAaV8uo7eRnFgALAMaPH5+yvHZU29jSXisIgsLBxhYA+uflcNqY\nocydWUZFWSHTygopLxmMiMR/wZ0vwZYVcMqnk1QCY0wogg7pxmpXgyg7O9z8dCO0ACEi+cCjwLdV\n9ZCI3AfcBah//inwtY4/p6qLgEUAM2bM0NTl2Hll+0Fuf3w922saAbfE++QR+Xz8lJFMKyukYlwh\nHxpdQP+8JHbv7PoHLPsSFI6Hi3+YvM81xqRe7FwIa2L6IBHphwsOy1T1MQBV3R/z/q+A5WHkrTvP\nbqjimw+9wbjCQXzv4lOYVjaM08cOo2BgH67jvnslLPsiDB0D85+M9D8mY8wJKPA1iIPboK3Jmphi\niWtn+Q2wUVV/FpNe6vsnAD4HvJnqvHXnj6v2cOtj6zhjXCG/vXYmRUNSMJN5zyr43RdclXT+k8f/\nYRlj0lewYF/VOn8c3f/XYdQgzgWuAdaLyBqfdjtwlYhU4JqYdgL/FELePkBVuf9v27n7L5v42JTh\n/NdXzmLIgBT82va+Br/7vJthee1yGFra99c0xvS9ISMAgcq17jiis6ghnFFMLwGd9dY+neq89OTY\nMeWHf9nIr/6+g89MG8NPvzgtuX0LXdn3Bjz4ORhcDPOXu+YlY0xmyM1zTcVV691xhGsQNpO6C61H\nj/Evj6zjV3/fwbxZJ/HzuRWpCQ6Va+GBz8LAYS44DEv7EcDGmI7yR7mNgyCyC/WBBYhONbce5esP\nvsajr7/Ddz5xMndedho5OQkMUT1RVevhgcthQIELDoVxzpcwxkRbsC8E4mZSR1Q053eHqL6pleuX\nrmL1rlru+uxUrjnnpNRceP8GWHoZ9BviOqSLUnRdY0zqBc1KQ4ZHdpkNsADxPtWHmpm3+FW21TTw\ni6vO5NIzUtT2X73RBYe8gTD/CSiekJrrGmPCEcyFiHDzEliAaLfrYCPX/OZVDjQcYfG1M/nYlBSN\nLKjZDEs/49aIv3Y5lExKzXWNMeEJhqxHeAQTWIAAYMO+euYvXsXRY8f4/Q3nUFFWmJoLH9jigoPk\nWHAwJpsEcyEiPIIJLECwcvtBrl+6mvyBefxhwSwmjyxIzYUPboMll7r9Ha59CoZPSc11jTHha29i\nshpEZK14az/f+P3rjCsaxIPXnR3/Mty99e52FxyOtbmag20AZEx2KbAaRKT9cfUebntsPVPHDmNJ\nqpbOAHh3Byz5DLQ1u+Bg24cak32GjYXLfgEnXxJ2TrqVlQHi/7Ye4LuPrOvbpTOa6tz+0bU7XVAI\nXleuc0vAzn8SRp2W/OsaY9LD9Hlh56BHWRkgZk0s4d8uP40rZ46Pf3b0saNwaF+HILDTHb+74/gs\nycDg4W746skXwUe/BaOnJloMY4zpU1kZIHJq3mLe61+D1+P8gLYjcGgvHG2J+dA8GFbmgsDUM6Fo\ngntdVO4eA1LU+W2MMUmSlQGCvIGJdQzn5MGpl7kgUFTuAsHQcZGeEWmMMb2VnX/RSibBlx4IOxfG\nGBNptlifMcaYTlmAMMYY06nIBQgRuVhENovIVhG5Nez8GGNMtopUgBCRXOCXwCXAqbhtSE8NN1fG\nGJOdIhUggI8AW1V1u6q2AH8ALg85T8YYk5WiFiDGAntijt/xacYYY1IsagGiRyKyQERWi8jqmpqa\nsLNjjDEZK2oBYi8QuxHzOJ/WTlUXqeoMVZ0xYkS0l8o1xph0Jqoadh7aiUge8DYwBxcYVgFfVtUN\nXZxfA+yK83LDgQNx/my6sjJnBytzdkikzCepao/fsCM1k1pV20TkG8CzQC6wuKvg4M+PuwohIqtV\ndUa8P5+OrMzZwcqcHVJR5kgFCABVfRp4Oux8GGNMtotaH4QxxpiIyOYAsSjsDITAypwdrMzZoc/L\nHKlOamOMMdGRzTUIY4wx3cjKAJGJCwKKSJmIvCgib4nIBhG5yacXi8gKEdnin4t8uojIvf53sE5E\npodbgviJSK6IvCEiy/3xBBFZ6cv23yLS36cP8Mdb/fvlYeY7XiJSKCKPiMgmEdkoIrMy/T6LyHf8\nv+s3ReQhERmYafdZRBaLSLWIvBmT1uv7KiLz/flbRGR+InnKugCRwQsCtgG3qOqpwDnAjb5ctwLP\nq+oU4Hl/DK78U/xjAXBf6rOcNDcBG2OOfwTco6qTgVrgOp9+HVDr0+/x56WjnwPPqOopwDRc2TP2\nPovIWOBbwAxVnYobAn8lmXeflwAXd0jr1X0VkWJgIXA2bm27hUFQiYuqZtUDmAU8G3N8G3Bb2Pnq\ng3L+GfgksBko9WmlwGb/+n7gqpjz289Lpwdutv3zwGxgOSC4yUN5He83bn7NLP86z58nYZehl+Ud\nBuzomO9Mvs8cX6Ot2N+35cBFmXifgXLgzXjvK3AVcH9M+vvO6+0j62oQZMGCgL5KfSawEhilqpX+\nrSpglH+dKb+H/wC+CxzzxyVAnaq2+ePYcrWX2b9f789PJxOAGuC3vlnt1yIyhAy+z6q6F/gJsBuo\nxN2318js+xzo7X1N6v3OxgCR0UQkH3gU+LaqHop9T91XiowZtiYilwLVqvpa2HlJoTxgOnCfqp4J\nNHK82QHIyPtchFv2fwIwBhjCB5tiMl4Y9zUbA0SPCwKmKxHphwsOy1T1MZ+8X0RK/fulQLVPz4Tf\nw7nAZSKyE7d3yGxc+3yhX9cL3l+u9jL794cBB1OZ4SR4B3hHVVf640dwASOT7/MngB2qWqOqrcBj\nuHufyfc50Nv7mtT7nY0BYhUwxY+A6I/r7Hoi5DwlTEQE+A2wUVV/FvPWE0AwkmE+rm8iSJ/nR0Oc\nA9THVGXTgqrepqrjVLUcdx9fUNWrgReBK/xpHcsc/C6u8Oen1TdtVa0C9ojIh3zSHOAtMvg+45qW\nzhGRwf7feVDmjL3PMXp7X58FLhSRIl/zutCnxSfsTpmQOoI+hVs1dhvwr2HnJ0llOg9X/VwHrPGP\nT+HaXp8HtgD/AxT78wU3mmsbsB43QiT0ciRQ/guA5f71ROBVYCvwMDDApw/0x1v9+xPDznecZa0A\nVvt7/SegKNPvM3AnsAl4E3gQGJBp9xl4CNfH0oqrKV4Xz30FvubLvhX4aiJ5spnUxhhjOpWNTUzG\nGGNOgAUIY4wxnbIAYYwxplMWIIwxxnTKAoQxxphOWYAwBhCRoyKyJubR7Sq/IvJ1EZmXhOvuFJHh\niX6OMX3BhrkaA4hIg6rmh3Ddnbgx7AdSfW1jemI1CGO64b/h/7uIrBeRV0Vksk+/Q0T+2b/+lrh9\nONaJyB98WrGI/MmnvSIiZ/j0EhF5zu9t8GvchKfgWl/x11gjIveL2+ciV0SW+H0Q1ovId0L4NZgs\nZQHCGGdQhyamuTHv1avq6cB/4laP7ehW4ExVPQP4uk+7E3jDp90OPODTFwIvqeppwOPAeAAR+TAw\nFzhXVSuAo8DVuFnTY1V1qs/Db5NYZmO6ldfzKcZkhSb/h7kzD8U839PJ++uAZSLyJ9zSF+CWPvkC\ngKq+4GsOQ4Hzgc/79KdEpNafPwc4C1jllhtiEG5htieBiSLyC+Ap4Ln4i2hM71gNwpieaRevA5/G\nrYszHfcHPp4vXgIsVdUK//iQqt6hqrW4XeP+iqud/DqOzzYmLhYgjOnZ3Jjnl2PfEJEcoExVXwS+\nh1taOh/4O66JCBG5ADigbn+OvwFf9umX4BbaA7cg2xUiMtK/VywiJ/kRTjmq+ijwfVwQMiYlrInJ\nGGeQiKyJOX5GVYOhrkUisg44gtvSMVYu8DsRGYarBdyrqnUicgew2P/cYY4v2Xwn8JCIbAD+gVvK\nGlV9S0S+Dzzng04rcCPQhNs9Lvgyd1vyimxM92yYqzHdsGGoJptZE5MxxphOWQ3CGGNMp6wGYYwx\nplMWIIwxxnTKAoQxxphOWYAwxhjTKQsQxhhjOmUBwhhjTKf+H1oADC/5psv7AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nwD41YstHyGx"
      },
      "source": [
        "### Save model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YU3N0zE_HxQb",
        "colab": {}
      },
      "source": [
        "agent.model_prediction.save('dqn_1.model')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9ILbxFFfIckN"
      },
      "source": [
        "### Rendering Test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iFZ1eIgjHwXw",
        "colab": {}
      },
      "source": [
        "done = False\n",
        "state = env.reset()\n",
        "while not done:\n",
        "    qs_list = agent.get_qs(state)\n",
        "    action = np.argmax(qs_list())\n",
        "    next_state, _, done, _ = env.step(action)\n",
        "    state = next_state\n",
        "    env.render()\n",
        "\n",
        "env.close()\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}